https://modelscope.github.io/twinkle-web/zh/docs/components/loss-scale/损失缩放HugoBlox Kit (https://hugoblox.com)zh-Hanshttps://modelscope.github.io/twinkle-web/media/logo_hu_fedc6a0bfe689b18.pnghttps://modelscope.github.io/twinkle-web/zh/docs/components/loss-scale/https://modelscope.github.io/twinkle-web/zh/docs/components/loss-scale/lossscale/Mon, 01 Jan 0001 00:00:00 +0000https://modelscope.github.io/twinkle-web/zh/docs/components/loss-scale/lossscale/<p>LossScale 组件控制训练过程中的损失缩放，确保数值稳定性，在混合精度训练中尤为重要。</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">twinkle.loss_scale</span> <span class="kn">import</span> <span class="n">LossScale</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">loss_scale</span> <span class="o">=</span> <span class="n">LossScale</span><span class="p">()</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># 在反向传播前对损失进行缩放</span> </span></span><span class="line"><span class="cl"><span class="n">scaled_loss</span> <span class="o">=</span> <span class="n">loss_scale</span><span class="p">(</span><span class="n">loss</span><span class="p">,</span> <span class="n">num_tokens</span><span class="p">)</span> </span></span></code></pre></div><p>LossScale 通过有效 token 数量对损失值进行归一化，确保不同批次大小和序列长度下梯度幅度的一致性。</p> <blockquote class="border-l-4 border-neutral-300 dark:border-neutral-600 pl-4 italic text-neutral-600 dark:text-neutral-400 my-6"> <p>LossScale 在模型训练流水线中内部使用。使用 <code>model.forward_backward()</code> 时会自动应用。</p> </blockquote>