Colors work by which parts of the light spectrum get absorbed and which get reflected. A shirt that absorbs most wavelengths, but largely reflects green, will look green.

And in that same vein, colors will appear darker, if less light is reflected in general. So, a white shirt may reflect 90+% of light, whereas a black shirt may absorb 90+% of light.

And ultimately, absorption means that the light's energy is turned into warmth. So yeah, a darker shirt heats up more in the sun.