I know it's not zero and that's where the offset is, but my point is it ought to be zero and there shouldn't be any offset. A scale doesn't start at 12, numbers have a meaning.

Using zero makes far more sense not just in how am/pm is indicated, but in how we actually think about time. Nobody considers 00:45 = 12:45am as part of the previous day since it's clearly after midnight. Using 12am is just an outdated convention established by people who counted hours only on a per-day basis and to whom the concept of a "zeroth hour" seemed irrational.