That completely stopped working for me a while ago, but Samsung’s Reminders app was able to do it well.

I started using it instead, and the other day, uninstalled the Google Assistant app - the only time I was opening it was by accident, when I’d drift off to sleep holding my phone, and he rudely awoken by the noise it made.