You might try selecting Tools > Options > Tagging Options, the put a check in "Find references incrementally" and '"Go To Reference" only lists references'. The second setting is particularly useful since it displays the list of files without then opening each to locate the exact reference.
If you have a large number of tags, you should also increase "Tagging cache size". The default is 32000k. For large projects, a value of 128000 may produce better performance. You may have to experiment to find the value that performs best for you. As I'm sure you are aware, it's possible to have a cache that is too large.
--Scott
