you really just need some matrix multiplication and calculating derivatives-- non-math-major college math type stuff.

Well sure you don't need a math degree for that, but most people really need to put some time into those topics. I.e., that kind of math is complex enough to constitute a barrier to entry into the field, particularly people with no free time to self-study or money for school.

Said differently: matrix math and basic calculus is hard, just not for you and I.