Perhaps looking down is interpreted by alpha males as a sign of submission, possibly triggering aggressivity too.

You never know with animals. Cats for example hate direct eye contact too - but only when you do it, so you'd never guess 🙂