Another annotation problem I've found has to do with the tool tip.   
In a lot of cases, when moving the mouse over the gutter icon for a particular annotation, the pop up contains the information for the wrong annotation.   Usually its from an annotation in a completely different file.   
The only thing I've noticed is that if I open the list of annotations the problem goes away and the pop up contains the right information.  This lasts for a while and then reverts to the wrong data again.   Other than this I haven't had much time to research the 'why' behind the problem.
Thanks -andy