Both Qwen3-Next and Kimi Linear use a 3:1 ratio, meaning for every three transformer blocks employing the linear Gated DeltaNet variant, there’s one block that uses full attention, as shown in the figure below.

<imgsrc="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gated_deltanet/01.webp"alt="Qwen3-Next versus Kimi Linear"style="zoom:47%;" />

<imgsrc="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gated_deltanet/01.webp"alt="Qwen3-Next versus Kimi Linear">

Gated DeltaNet has a gate similar to the gate in gated attention discussed earlier, except that it uses a SiLU instead of logistic sigmoid activation, as illustrated below. (The SiLU choice is likely to improve gradient flow and stability over the standard sigmoid.)

<imgsrc="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gated_deltanet/02.webp"alt="Gated DeltaNet"style="zoom:47%;" />

<imgsrc="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gated_deltanet/02.webp"alt="Gated DeltaNet"width=500px>

However, as shown in the figure above, the "gated" in the Gated DeltaNet also refers to several additional gates:

<imgsrc="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gated_deltanet/03.webp"alt="Quadratic attention"style="zoom:67%;" />

<imgsrc="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gated_deltanet/03.webp"alt="Quadratic attention"width=500px />

In Gated DeltaNet, there's no*n*-by-*n* attention matrix. Instead, the model processes tokens one by one. It keeps a running memory (a state) that gets updated as each new token comes in. This is what's implemented as, where`S` is the state that gets updated recurrently for each time step*t*.

Note that the above computes the`head_dim` as`emb_dim / n_heads`. I.e., 2048 / 16 = 128.

<imgsrc="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gated_deltanet/plot.webp"alt="Gated DeltaNet scaling"style="zoom:47%;" />

<imgsrc="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gated_deltanet/plot.webp"alt="Gated DeltaNet scaling"width=500px>