Rectified Linear Unit (ReLU)

𝑓(𝑧) = 𝑚𝑎𝑥(𝑧, 0)

• ✔ simple
• ✔ can wipe the negative signal out
• ✘ zero gradient for negative inputs, thus can be fragile during training and “die”

Leaky ReLU

𝑓(𝑧) = 𝑚𝑎𝑥(𝑧, 𝛼𝑧)

• ✔ non-zero gradient for negative inputs
• ✘ slope 𝛼 needs to be hand-tuned
• ✘ cannot wipe the negative signal out

Parametric ReLU (PReLU)

𝑓(𝑧) = 𝑚𝑎𝑥(𝑧, 𝛼𝑧) # is Leaky ReLU where 𝛼 is learned

• ✔ non-zero gradient for negative inputs
• ✘ cannot wipe the negative signal out

Scaled Exponential Linear Unit (SELU)

$f(z)=\{\begin{array}{ll}z& \text{if }x\ge 0\\ \alpha (e^z-1)& \text{if }x<0\end{array}$

• ✔ non-zero gradient for negative inputs
• ✘ 𝛼 needs to be hand-tuned
• ✘ exponential is computationally expensive

Exponential Linear Unit (ELU)

Is SeLU where 𝛼 = 1

Gaussian Error Linear Unit (GeLU)

𝑓(𝑧) = 𝑧 ⨯ 𝛷(𝑧)

where:

• 𝛷(𝑧) is the CDF of the Standard Gaussian
• $𝛷(z)=\frac{1}{2\pi }{\int }_{-\infty }^z{e}^{-\frac{t^2}{2}}dt$
• ✔ non-zero gradient for negative inputs
• ✘ is computationally expensive

SeLU

TODO

Concatenated ReLU (CReLU)

• has two outputs concatenated together (this DOUBLES the output dimension):
  • one normal ReLU
  • one negative ReLU
• In other words:
  • for positive input 𝑧 it outputs the following two values {𝑧, 0}
  • for negative input 𝑧 it outputs the following two values {0, 𝑧}

ReLU-6

𝑓(𝑧) = 𝑚𝑎𝑥(𝑚𝑖𝑛(𝑧, 6), 0)

• is ReLU capped at 6
• according to the authors, the upper bound encouraged their model to learn sparse features earlier