Sigmoid units - Artificial intelligence
Remember that the function inside units take as input the weighted sum, S, of the values coming from the units associated to it. The function inside sigmoid units calculates the following value that given a real-valued input S:
Where e is the base of natural logarithms, e = 2.718...
When we plot the output from sigmoid units given several weighted sums as input, it seems remarkably like a step function:
Certainly, getting a differentiable function which seems like the step function was the main point of the exercise. Definitely, not only is this function differentiable, but the derivative is fairly expressed simply in terms of the function itself:
Notice that the output values for the σ function range between but never make it to 1and 0. This is because e-S is never -ve, and the denominator of the fraction tends to 0 as S gets very large in the -ve direction, and tends to 1 as it gets very large in the +ve direction. This tendency happens quickly: the middle ground between 0 and 1 is hardly ever seen because of the sharp (near) step in the function. Because of it seems like a step function, we may think of it firing and not-firing as in a perceptron: if a positive real is input, the output will normally be close to +1 and if a negative real is input the output will normally be close to -1.