Open PhonePe app → Profile → Settings → Smart Assistant → Turn off smart speaker. You can also revoke microphone permissions in phone settings if you want complete control.
