Assuming the phone's security works as intended, what you're saying is true. However, it's a legit concern that the security is not airtight, and physical access is not actually required to harvest your biometric data.

I know the phone manufacturers make all sorts of claims about how secure biometric data is, but they have a profit motive to do so. I'm not being brick-up-my-windows paranoid by pointing out all the security failures and breaches we've seen over the years. Companies that have billions on the line are still frequently falling short at securing their own assets, much less their customer's data.

I understand biometrics are convenient, and many folks love the ease / coolness factor of using them. Just don't kid yourself that it's secure by requiring your physical phone. Once the dark web has a digital copy of your biometric data, it's compromised forever.