一起读《动手学深度学习（PyTorch版）》- 多层感知机 - 激活函数
<div class='showpostmsg'><p>多层感知机，隐藏层和输出层之间的全连接层如果全部是线性相关的话，那么多层感知机可以等价于单层感知机，因为线性关系的堆积仍然是线性的。激活函数的作用在于引入非线性因素，防止多层感知机退化为线性模型。</p><p> </p>
<article data-content="[{"type":"block","id":"f9TB-1729522105337","name":"paragraph","data":{},"nodes":[{"type":"text","id":"IabC-1729522105336","leaves":[{"text":"多层感知机中，隐藏层和输出层之间的全连接层如果全部是线性相关的话，那么多层感知机可以等价于单层感知机，因为线性关系的堆积仍然是线性的。激活函数的作用在于引入非线性因素，防止多层感知机退化为线性模型。","marks":[]}]}],"state":{}}]">
<article data-content="[{"type":"block","id":"hsM6-1729522310892","name":"heading","data":{"level":"h3","style":{}},"nodes":[{"type":"text","id":"1hV7-1729522310891","leaves":[{"text":"ReLU激活函数（Rectified linrear unit）","marks":[]}]}],"state":{}},{"type":"block","id":"IybH-1729522930419","name":"paragraph","data":{},"nodes":[{"type":"text","id":"nsws-1729522930417","leaves":[{"text":"负数时导数为0，正数时导数为1","marks":[]}]}],"state":{}}]">
<p>ReLU激活函数（Rectified linrear unit）</p>
<p>负数时导数为0，正数时导数为1</p>
<pre>
<code>import torch
import torchvision
from torch.utils import data
from torchvision import transforms
import matplotlib.pyplot as plt
from torch import nn
x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True)
y = torch.relu(x)
plt.plot(x.detach(), y.detach(), 'x', 'relu(x)')
plt.show()</code></pre>
<p> </p>
</article>
<pre>
<code>y.backward(torch.ones_like(x), retain_graph=True)
plt.plot(x.detach(), x.grad, 'x', 'grad of relu')</code></pre>
<p> </p>
<article data-content="[{"type":"block","id":"MtyP-1729523078470","name":"heading","data":{"level":"h3","style":{}},"nodes":[{"type":"text","id":"gCwU-1729523078468","leaves":[{"text":"sigmoid（squashing function）","marks":[]}]}],"state":{}},{"type":"block","id":"xsVc-1729695509122","name":"paragraph","data":{},"nodes":[{"type":"text","id":"x0bg-1729695509121","leaves":[{"text":"sigmoid函数，将输入变换为区间(0, 1)上的输出，通常称为挤压函数（squashing function），低于0就是0，高于1就是1","marks":[]}]}],"state":{}},{"type":"block","id":"hQRw-1729695308666","name":"paragraph","data":{},"nodes":[{"type":"text","id":"1MOR-1729695308664","leaves":[{"text":"当输入接近0时，sigmoid函数接近线性变换","marks":[]}]}],"state":{}}]">
<p>sigmoid（squashing function）</p>
<p>sigmoid函数，将输入变换为区间(0, 1)上的输出，通常称为挤压函数（squashing function），低于0就是0，高于1就是1</p>
<p>当输入接近0时，sigmoid函数接近线性变换</p>
<pre>
<code>import torch
import torchvision
from torch.utils import data
from torchvision import transforms
import matplotlib.pyplot as plt
from torch import nn
if torch.cuda.is_available():
device = torch.device("cuda")
x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True, device=device)
y = torch.sigmoid(x)
y=y.to(device)
plt.plot(x.to("cpu").detach(), y.to("cpu").detach(), '.', 'sigmoid(x)')
plt.show()</code></pre>
<p> </p>
<article data-content="[{"type":"block","id":"cczo-1729695503085","name":"heading","data":{"level":"h3","style":{}},"nodes":[{"type":"text","id":"FhGr-1729695503084","leaves":[{"text":"tanh(双曲正切)","marks":[]}]}],"state":{}},{"type":"block","id":"PKrV-1729696266905","name":"paragraph","data":{},"nodes":[{"type":"text","id":"fhlU-1729696266904","leaves":[{"text":"-1,1的范围","marks":[]}]}],"state":{}}]">
<p>tanh(双曲正切)</p>
<p>-1,1的范围</p>
<p> </p>
</article>
</article>
</article>
</div><script> var loginstr = '<div class="locked">查看本帖全部内容，请<a href="javascript:;" style="color:#e60000" class="loginf">登录</a>或者<a href="https://bbs.eeworld.com.cn/member.php?mod=register_eeworld.php&action=wechat" style="color:#e60000" target="_blank">注册</a></div>';
if(parseInt(discuz_uid)==0){
(function($){
var postHeight = getTextHeight(400);
$(".showpostmsg").html($(".showpostmsg").html());
$(".showpostmsg").after(loginstr);
$(".showpostmsg").css({height:postHeight,overflow:"hidden"});
})(jQuery);
} </script><script type="text/javascript">(function(d,c){var a=d.createElement("script"),m=d.getElementsByTagName("script"),eewurl="//counter.eeworld.com.cn/pv/count/";a.src=eewurl+c;m.parentNode.insertBefore(a,m)})(document,523)</script> <p>楼主分享的技术内容值得收藏学习，希望楼主继续分享更多内容</p>
<p>防止多层感知机退化为线性模型这个有点难度</p>
<p>激活函数对于多层感知机来说是不是就相当于中断，打破多层界限？</p>
秦天qintian0303 发表于 2024-10-24 08:37
激活函数对于多层感知机来说是不是就相当于中断，打破多层界限？
<p>不是</p>
chejm 发表于 2024-10-23 23:54
楼主分享的技术内容值得收藏学习，希望楼主继续分享更多内容
<p>好的</p>
页:
[1]