Example calculation of entropy, Computer Engineering

Assignment Help:

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.


Related Discussions:- Example calculation of entropy

Address fields and phone fields , Make a nested struct to store a person's ...

Make a nested struct to store a person's name, address, and phone numbers.  The struct should have 3 fields for the name, address, and phone. The address fields and phone fields wi

Write a short note on pointer operators in c, Write a short note on pointer...

Write a short note on pointer operators in c Pointers (that is, pointer values) are generated with the ''address-of'' operator &, which we can also think of as the ''pointer-to

What is parallel sections construct, What is Parallel Sections Construct? ...

What is Parallel Sections Construct? The parallel sections construct is a shortcut for specifying a parallel construct comprising one sections construct however no other statem

Is strategy - information system, IS strategy - Information System As ...

IS strategy - Information System As an operations manager when attempting to navigate your organisation through the information landscape and make best use of the available sy

Where can i find conference information, Georg Thimm handles a webpage that...

Georg Thimm handles a webpage that lets you search for upcoming or past conferences in a range of AI disciplines.

How can we draw a circle with gimp, Ans) The simplest way is to make a new ...

Ans) The simplest way is to make a new selection with Ellipse Select tool and stroke it (Edit -> Stroke Selection...). We welcome patches that add tools to draw geometric primitive

What is presentation layer, Q. What is Presentation layer? Presentatio...

Q. What is Presentation layer? Presentation layer: When two hosts are communicating with each other they might use different coding standards and character sets for represent

What is data in computers, Q. What is data in computers? In modern digi...

Q. What is data in computers? In modern digital computers data is signified in binary form by employing two symbols 0 and 1. These are known as binary digits or bits however da

Explain the working of a 3-bit D/A converter, With the help of R-2R binary ...

With the help of R-2R binary network, explain the working of a 3-bit D/A converter and derive an expression for the output voltage. Ans: We see a 3-bit R-2R ladder D/A netwo

Show the process of message passing, Q. Show the process of message passing...

Q. Show the process of message passing? The subsequent issues are determined by system in process of message passing: 1)  Whether receiver is prepared to receive message

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd