As someone who has taught both mathematics and statistics in his life the real difference boils down to proof Vs evidence.

The mathematician is uncertain because Gödel showed no system can prove its own consistency. Proof is (generally) rigourous enough that this is the main issue; once it has been proven (assuming your axioms are good), it's considered true.

The statistician is uncertain because they work with samples rather than the population. There is also the issue of inferring causation even if your sample isn't unrepresentative. With statistics you're always building evidence, but you can never have concrete proof via statistical methods alone.

Also fun fact, given that there is more than one type of mathematics (e.g. platonist Vs intuitionistic), some giving different answers to the same question (excluded middle/trichotomy on the reals), and all of which are equiconsistent, we realise that mathematics really is just a branch of philosophy (i.e. what axioms are you willing to believe).