Rebuilding the adversarial KataGo testbench, Part 3

After finishing the sweep in Part 2, I got a pretty clean result: the adversary scored zero wins. The "victim", the strongest checkpoint at the time of writing, never really gave it any chance. Instead of dragging things out to maximize score, it usually settled the ko quickly and went for the clean capture. Overall, I was happy to see the latest version of KataGo going cutthroat against its old adversary.

Interestingly, the adversary did manage to recreate cyclic scenarios in every game I reviewed manually. One thing I learned from the original work is that they did not specifically train the adversary to exhibit cyclic attacks; that behavior seems to emerge from the AMCTS training process itself. So maybe if we train AMCTS against the strongest checkpoint instead, a new adversary could uncover different blind spots and swing the games back in its favor.

At first, I was a little worried about the low visit settings in my setup. If the outcome had been less one-sided, I would have had doubts about my AMCTS port into the latest KataGo engine. But with the newer checkpoint winning this cleanly, I think it is fair to say that the newer engine does include the low-visit LCB fix that helps lessen this particular weakness. To be fair, the paper also verified the victim at 10^6 and 10^7 visits.

So even though this particular adversary no longer works against the newer checkpoint, I do not think the broader line of attack is dead yet. Perhaps we are still a long way from the optimal minimax solution. But I am hopeful that the adversarial nature of self-play will eventually get us there.

The benchmark code is available on GitLab.