I agree that we need far stronger admin and moderation tools to fight spam and bots. I disagree with the idea of a whitelist approach, and think taking even more from email (probably the largest federated system ever) could go a long way.

With email, there is no central authority granting “permission” for me to send stuff. There are technologies like SPF, DKIM, DMARC, and FcRDNS, which act as a minimum bar to reach before most servers trust you at all, then server-side spam filtering gets applied on top and happens at a user, domain, IP, and sometimes netblock level. When rejections occur, receiving servers provide rejection information, that let me figure out what is wrong and contact the admins of that particular server. (Establish a baseline of trust, punish if trust is violated)

A gray-listing system for new users or domains could generate reports once there is a sufficient amount of activity to ease the information gathering an admin would have to do in order to trust a certain domain. Additionally, I think establishing a way for admins to share their blacklisting actions regarding spam or other malicious behavior (with verifiable proof) could achieve similar outcomes to whitelisting without forcing every instance operator to buy in to a centralized (or one of a few centralized) authority on this. This would basically be an RBL (which admins could choose to use) for Lemmy. This could be very customizable and allow for network effects (“I trust X admin, apply any server block they make to my instance too” sort of stuff).

I think enhancements to Lemmy itself would also address help. Lemmy itself could provide a framework for filtering and report when an instance refuses a federated message with relevant information, allowing admins to make informed decisions (and see when there are potential problems on their instance). Also having ways to attach proof of bad behavior to federated bans at an instance level, and some way to federate bans (again with proof) from servers that aren’t a user’s home instance.

Finally, as far as I can tell everything following a “Web of Trust” model (basically what you are proposing) has struggled to gain widespread adoption. I have never been to a key signing party. I once made a few proofs on keybase, but that platform never really went anywhere. This doesn’t mean your solution won’t work, it just concerns me a little.

I expanded a bit more on some of how email tooling could be used within lemmy in this comment as well. My ideas aren’t fully baked yet, but I hope they at least make some sense.