I added a bunch of dsay to see what was going on. Looked like callback2 wasn't getting called as often as it should. Then I added more dsay() and the issue went away and seems to work now - xretrace_cursor_steps doesn't ask to start xretrace and it does seem to navigate to the right spots. Some kind of race condition and I can't get it back.
I had both of these happen before, xretrace_cursor_steps() continually asking to reload, then when it finally does work the <left>, <right> go to the right places, but then eventually the <left>, <right> go to the wrong places. It is working now, even after I removed some of my dsay that made it work. Not sure how I can get it to repro again.