TD-leaf and another negamax bug

TD-lambda and TD-leaf are separated by the observation that minimax with a parameterised leaf heuristic is differentiable* in the parameters. "All" that had to be done was to extract the "principal variation": which one position survived all of the pruning to become the reported value.

This was a bit fiddly and uncovered a subtle bug (again) in negamax.c: we were not being complete in our overriding of the TT entry and by omitting key it was possible that later searches would find incorrect values.

The more interesting question to me was: how do we introduce enough variety while not corrupting the PV chain? I settled on having selfplay have one of the players be epsilon-random but the other greedy (minimax) and simply report only the greedy portion PV chain. This appears to work well enough in practice and doesn't introduce nonsense training data for the net (we don't force its actions to be nonsensical). I also deciced, perhaps in a departure from tradition, that we had good supervised labels for PVs that happened to be winning states, so the training overrides the targets in that case. Finally rolled back some of the ambition in nn_train.py default arguments.

*almost always

1 files changed, 8 insertions, 10 deletions

diff --git a/include/actions.h b/include/actions.h
index 913a8cb..d96033c 100644
--- a/include/actions.h
+++ b/include/actions.h
@@ -54,24 +54,22 @@ extern int8_t move_deltas[4];
 // Methods
 // ===================================================================
 
+#define AL_MAX_LEVELS 8
 void actions_init(const uint8_t board_size);
 
-/*
- * Generate the legal actions. The move list is drawn from a static
- * per-search-level scratch buffer: one buffer per recursion level,
- * so nested generate calls at different levels do not clobber each
- * other. `level' is the recursion depth within the current search
- * (0 at the root). The returned pointer is valid until the next
- * generate call at the same level.
- */
+// Generate the legal actions. The move list is drawn from a static
+// per-search-level scratch buffer: one buffer per recursion level, so nested
+// generate calls at different levels do not clobber each other. `level' is the
+// recursion depth within the current search (0 at the root). The returned
+// pointer is valid until the next generate call at the same level.
 action_t *actions_generate_at(tak_state_p state, uint32_t *num_actions,
                               const uint32_t level);
 
 action_t *actions_generate(tak_state_p state, uint32_t *num_actions);
 char action_in_list(action_t action, action_t *actions,
                     const uint32_t num_actions);
-// Move `action' to the front of the list. Returns EXIT_FAILURE (and
-// leaves the list untouched) if the action is not in the list.
+// Move `action' to the front of the list. Returns EXIT_FAILURE (and leaves the
+// list untouched) if the action is not in the list.
 int action_move_to_front(action_t action, action_t *actions,
                          const uint32_t num_actions);