Over fitting Considerations - artificial intelligence

Left  unexamined ,  back  propagation  in  multi-layer  networks  may  be very susceptible  to over fitting itself to the training examples. The following graph plots the error on the training and test set as the number of weight updates increases. It is error prone of networks left to train unchecked.

Alarmingly, even though the error on the training set continues to slowly decrease, the error on the test set essentially begins to increase towards the end. It is clearly over fitting, and it relates to the network starting to find and fine-tune to idiosyncrasies in the data, rather than to general properties. Given this phenomena, it would not be wise to use some sort of threshold for the error as the termination condition for back propagation.

In the cases where the number of training examples is high, one antidote to over fitting is to crack the training examples into a set to use to train the weight and a set to hold back as an internal validation set. This is a mini-test set, which may be used to keep the network in check: if the error on the validation set reaches minima and then start to increase, then it could be over fitting in beginning to occur.

Note that (time permitting) it is good giving the training algorithm the advantage of the doubt as much as possible. That is, in the validation set, the error may also go through local minima, and it is unwise to stop training as soon as the validation set error begin to increase, as a better minima can be achieved later on. Of course, if the minima are never bettered, then the network which is in final presented by the learning algorithm should be re-wound to be the 1 which produced the minimum on the validation set.

Another way around over fitting is to decrease each weight by a little weight decay factor during each epoch. Learned networks with large (negative or positive) weights tend to have over fitted the data, because larger weights are needed to accommodate outliers in the data. Thus, keeping the weights low with a weight decay factor can help to steer the network from over fitting.

Posted Date: 10/2/2012 6:39:54 AM | Location : United States

Ask an Expert

Related Discussions:- Over fitting considerations - artificial intelligence, Assignment Help, Ask Question on Over fitting considerations - artificial intelligence, Get Answer, Expert's Help, Over fitting considerations - artificial intelligence Discussions

Write discussion on Over fitting considerations - artificial intelligence
Your posts are moderated

Write your message here..

Related Questions

What is machine.config, What is Machine.config?  Machine configuration ... What is Machine.config?  Machine configuration file: The machine. config file contains settings that apply to the whole computer.  The machine.config, which can be found in the

What is computer communications networks, Q. What is Computer communication... Q. What is Computer communications networks? Computer communications networks are the outcome of a combination of computers and telecommunication products. An interconnected gr

What is instruction register, Q. What is Instruction Register? Instruct... Q. What is Instruction Register? Instruction Register (IR): Here instructions are loaded before execution. Comments on figure above are as below: All representations are

I have a user defined function that does not recalculate, Contain all the c... Contain all the cells that your UDF depends on in the argument list. Or enter this as the first statement in your Function: Application.Volatile This will cause the functi

Neural architectures, Neural architectures are appealing as mechanisms for... Neural architectures are appealing as mechanisms for implementing intelligence for a number of reasons. Traditional AI programs tend to be brittle and overly sensitive to noise

Decoder, how to make a dec oder how to make a dec oder

When should we stop testing, When should you stop testing? It is quite ... When should you stop testing? It is quite complex to determine as many modern software applications are so complex and run in like an interdependent environment that complete t

Time complexity, Take a look at the code and try to understand the logic, p... Take a look at the code and try to understand the logic, particularly, the way scrambled words are disambiguated, i.e.., whether they are matched successfully against a correct wor

Extract data from tables without using the event ''get'', Is it possible to... Is it possible to extract data from tables without using the event 'GET' in the report with an appropriate LDB. False.   You can extract data from tables using Select stateme

What is ternary association, What is ternary association Associations c... What is ternary association Associations can be binary, ternary, or have higher order. In use, the vast majority of it is binary or ternary associations. Except a ternary assoc