This is a good point with respect to the inconsistency.
Unfortunately, the current implementation of the References tool window requires an instance for find-next / find-prev.
However, as a workaround, it will work with a pinned (auto-hidden) References tool window, so you could try that, and still use toggle-refs to make it visible when you want to see it.