Note that the`KV_cache_DeltaNet` memory size doesn't have a context length (`n_tokens`) dependency. Also, we have only the memory state S that we store instead of separate keys and values, hence`2 × bytes` becomes just`bytes`. However, note that we now have a quadratic`n_heads × d_head` in here. This comes from the state :

Note that the`KV_cache_DeltaNet` memory size doesn't have a context length (`n_tokens`) dependency. Also, we have only the memory state S that we store instead of separate keys and values, hence`2 × bytes` becomes just`bytes`. However, note that we now have a quadratic`d_head × d_head` in here. This comes from the state :