You're right that light is not "a wave and a particle depending on whether you observe it". Instead, light is a quantized field. It is a field because it exists at every point and allows for wave-like behavior such as superposition and interference (both things seen in all fields, like waves in water or radio, etc.). But it is quantized because when the field interacts it does so via photons which can only exist in integer quantities. This quantization of interaction of the underlying continuous field gives us all the "weirdness" we see. Okay, not quite all of it, there are still even weirder parts of quantum mechanics, but it does explain the double slit experiment.