Commit7d92267

authored

fix(GatedDeltaNet): Init param A from log of a uniform distrib (#906)

1 parent35354fa commit7d92267Copy full SHA for 7d92267

File tree

-1

lines changed

-1

lines changed

Lines changed: 2 additions & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -166,7 +166,8 @@ class GatedDeltaNet(nn.Module):`
`166`	`166`	`# A_log + W_alpha(x) + dt_bias`
`167`	`167`	`self.W_alpha= nn.Linear(d_in, num_heads,bias=False)`
`168`	`168`	`self.dt_bias= nn.Parameter(torch.ones(num_heads))`
`169`		`-self.A_log= nn.Parameter(torch.zeros(num_heads))`
	`169`	`+ A_init= torch.empty(num_heads).uniform_(0,16)`
	`170`	`+self.A_log= nn.Parameter(torch.log(A_init))`
`170`	`171`	`# We could implement this as`
`171`	`172`	`# W_alpha = nn.Linear(d_in, num_heads, bias=True)`
`172`	`173`	`# but the bias is separate for interpretability and`

Comments

(0)