use silu instead of relu with the intuition being that we avoid dead neurons

In a head-to-head this is +187 elo against the previous weights, although it trained more.

0 files changed, 0 insertions, 0 deletions