In many cases the key exchange (kex) for symmetric ciphers are done using slower asymmetric ciphers. Many of which are vulnerable to quantum algos to various degrees.

So even when attacking AES you’d ideally do it indirectly by targeting the kex.