An activation transforms z into the neuron’s output. Nonlinear activations let a network represent relationships that cannot be captured by one straight line or plane. A linear output is still useful for regression.

Function	Rule and range	Meaning / common use
Step	1 if z ≥ 0, otherwise 0; {0, 1}	A hard perceptron decision; not a smooth probability
Linear	f(z) = z; all real numbers	A continuous regression output
Sigmoid	1/(1 + e^−z); (0, 1)	One binary-class probability
Tanh	(e^z − e^−z)/(e^z + e^−z); (−1, 1)	Zero-centred outputs; can saturate
ReLU	max(0, z); [0, ∞)	Common hidden-layer activation
Leaky ReLU	z if z ≥ 0, otherwise αz	A small negative-side slope, e.g. α = 0.01
Softmax	pⱼ = e^zⱼ/Σₖe^zₖ	A vector of probabilities for mutually exclusive classes; sums to 1

e in an exponential is the mathematical constant approximately 2.718, not the prediction-error symbol. Sigmoid(0) = 0.5; tanh(0) = 0; ReLU(−3) = 0; ReLU(3) = 3; LeakyReLU(−3) = −0.03 when α = 0.01.

A ReLU unit can become inactive for all its training inputs and receive no useful gradient: the dying ReLU problem. Leaky ReLU allows a small gradient on the negative side. Sigmoid and tanh can have tiny derivatives at large positive or negative inputs; this contributes to vanishing gradients in deep networks.

For multiclass classification, softmax acts on all class scores together. Three independent sigmoid outputs do not generally add to 1.