Unfortunately leaf curl is inevitable and can only be managed by spraying. If this isn’t a preference yeah I agree time to get rid of the tree and plant something more robust. As for the lemon tree, my recommendation is to ignore any and all advice that says to cut off the galls. Spent a few years cutting off galls from my lemon tree, and the poor thing could never recover because it was getting cut back too much. Left it totally alone for a couple of years, just fertilised, and it is now lush and heaving with fruit. It is also full of galls but that seems to have zero impact on its fruit producing ability. There is enough tree for both me and the wasps to share.