I’ve collected 4 and sages and unlocked every tower. Now what to do next?

I heard that Hyrule castle is tough with phantoms. Should I hit it now or prepare something in advance?