Hi,

There might be a small bug here:

I think in the last line theexp should be divided bymin_timescale rather than multiplied, since it's inverse timescales. Usuallymin_timescale is 1 so it doesn't matter. But e.g. if you fixmax_timescale and changemin_timescale, the resulting inverse timescale corresponding tomax_timescale changes.

A simpler implementation could be roughly something like this:

inv_timescales = exp(-linspace(log(min_timescale), log(max_timescale), num_timescales))

and from this one you can derive the current implementation, except with division instead of multiplication. It can be even simpler with logspace but tf seems to have this function only as experimental.

Let me know if this makes sense.

Thanks a lot!