- sigmoid(x)
sigmoid(x)=11+e−xd(sigmoid(x))dx=ex(1+e−x)2=sigmoid∗(1−sigmoid) \begin{align} sigmoid(x) &= \frac{1}{1+e^{-x}} \\ \frac{d(sigmoid(x))}{dx} &= \frac{e^x}{(1+e^{-x})^2} \\ &= sigmoid*(1-sigmoid) \end{align}sigmoid(x)dxd(sigmoid(x))=1+e−x1=(1+e−x)2ex=sigmoid∗(1−sigmoid)
defsigmoid(x):return1/(1+np.exp(-x))defsigmoid_derivative(x):returnsigmoid(x)*(1-sigmoid(x))- Tanh，求导时用到了分式求导。两数和*两数差=平方差
tanh(x)=ex−e−xex+e−xd(tanh(x))dx=4exe−x(ex+e−x)2=1−tanh2 \begin{align} tanh(x) =& \frac{e^x-e^{-x}}{e^x+e^{-x}} \\ \frac{d(tanh(x))}{dx} =& \frac{4e^xe^{-x}}{(e^x+e^{-x})^2} \\ =& 1-tanh^2 \end{align}tanh(x)=dxd(tanh(x))==ex+e−xex−e−x(ex+e−x)24exe−x1−tanh2
deftanh(x):returnnp.tanh(x)deftanh_derivative(x):return1-np.tanh(x)**2- ReLU(x)
ReLU(x)=max(0,x)dReLU(x)dx={1,x>00,x<=0 \begin{align} ReLU(x) &= max(0,x) \\ \frac{dReLU(x)}{dx} &= \begin{cases} 1, x > 0 \\ 0, x <= 0 \end{cases} \end{align}ReLU(x)dxdReLU(x)=max(0,x)={1,x>00,x<=0
defrelu(x):returnnp.maximum(0,x)defrelu_derivative(x):return(x>0).astype(float)# x 是向量，以一维为例，可能是【0，5，3，-5，9】，x>0进行了一个判断，得到【flase,true……】就是布尔值，astype转化为了数值，0.0，1.0- PReLU/Leaky ReLU
f(x)={x,x>0αx,x<=0 f(x)= \begin{cases} x,x>0 \\ \alpha x,x<=0 \end{cases}f(x)={x,x>0αx,x<=0
解决ReLU负半轴梯度为零导致的神经元失活问题，但在X=0处，左右梯度不同，一阶导数发生跳变，二阶导数为无穷大，梯度不稳定
- Mish
f(x)=x∗tanh(softplus(x))softplus(x)=ln(1+ex) f(x)=x*tanh(softplus(x)) \\ softplus(x)=ln(1+e^x)f(x)=x∗tanh(softplus(x))softplus(x)=ln(1+ex)
既解决了x<0时神经元永久失活问题，也解决梯度平滑问题，消除“硬折角”问题
GELU
GELU(x)≈0.5x(1+tanh(2π(x+0.044715x3)))\mathrm{GELU}(x)\approx0.5x\left(1+\tanh\left(\sqrt{\frac{2}{\pi}}\left(x+0.044715x^3\right)\right)\right)GELU(x)≈0.5x(1+tanh(π2(x+0.044715x3)))
Swish
f(x)=x∗sigmoid(βx)=x1+e−βx f(x)=x*sigmoid(\beta x) \\ = \frac{x}{1+e^{-\beta x}}f(x)=x∗sigmoid(βx)=1+e−βxx
激活函数对比图
激活函数一阶导数
