I can see it being useful for admins on their own instances to detect brigades or vote manipulation coming from different servers, but I don’t like that they are public, because it could discourage people from voting honestly on topics where they could potentially be harassed for engaging

Well, if someone gets harassed for voting then the harasser should be banned, just as if you get harassed for any other reason. So it’s not any different than posts or comments in that way.