I think I’ve settled on the latter. Disagreement is maybe best communicated by the absence of an upvote? And downvotes work best when they signal something that is just off base, and while not reportable, is not appreciated at a broad cultural level.
Don’t think the moderators need to action every instance when a person or two thinks something doesn’t belong here. The down vote can take care of that. Moderators can focus on blatant issues.