From my experience, it works something like this:

The butt is pretty dark, so you're brain takes a little more time to make a coherent image out of it (think of it like averaging what you see over a short time). This is amplified by the fact that while the center of your eye is better at seeing color, it's worse at seeing in the dark.

So essentially you're tracking the butt (blue: worst color vision when dark) with the center of your eye (slow) which isn't seeing the movement of your phone (yet) while you see the white move immediately because it's brighter and not in the center. Normally you wouldn't notice the difference in speed, but when both effects are combined it is large enough to notice. It also works best in a dark room with the screen brightness on low, probably.

TLDR: you see slow when dark and blue and center