- Notifications
You must be signed in to change notification settings - Fork0
✅ Optimized Swish activation function, for neural networks
License
xyproto/swish
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
An optimized Swish activation function (Ramachandran, Zoph and Le, 2017), for neural networks.
The graphs above were drawn using the program incmd/graph, which usesgoterm.
First run:
goos: linuxgoarch: amd64pkg: github.com/xyproto/swishBenchmarkSwish07-8 200000000 8.93 ns/opBenchmarkSwish03-8 200000000 8.95 ns/opPASSok github.com/xyproto/swish5.391sgoos: linuxgoarch: amd64pkg: github.com/xyproto/swishBenchmarkSwish07-8 2000000000 0.26 ns/opBenchmarkSwish03-8 2000000000 0.26 ns/opPASSok github.com/xyproto/swish1.108sThe optimizedSwish function is34x faster than the one that usesmath.Exp, and quite a bit faster than my (apparently bad) attempt at a hand-written assembly version.
The average error (difference in output value) between the optimized and non-optimized version is+-0.0013 and the maximum error is+-0.0024. This is forx in the range[5,3]. See the program incmd/precision for how this was calculated.
0.000150.00001goos: linuxgoarch: amd64pkg: github.com/xyproto/swishBenchmarkSwishAssembly07-8 500000000 3.63 ns/opBenchmarkSwishAssembly03-8 500000000 3.65 ns/opBenchmarkSwish07-8 2000000000 0.30 ns/opBenchmarkSwish03-8 2000000000 0.26 ns/opBenchmarkSwishPrecise07-8 200000000 9.07 ns/opBenchmarkSwishPrecise03-8 200000000 9.25 ns/opPASSok github.com/xyproto/swish 11.100sI have no idea why the assembly version is so slow, but0.26 ns/op isn't bad for a non-hand-optimized version.
- Version: 1.3.0
- License: MIT
- Author: Alexander F. Rødseth <xyproto@archlinux.org>
About
✅ Optimized Swish activation function, for neural networks