You remember correctly. From Mr Scott's Guide to the Enterprise, p24:

The great success of Enterprise's historic five-vear mission brought other changes to Star Fleet procedure. To honor the ship and her crew, Star Fleet Command unanimously elected in 2212 to drop the individual ship emblem system employed since 2206 and then adopt the insignia of Enterprise [Command Division] as the official insignia of Star Fleet. Each individual's branch department would no longer be denoted by shirt color; rather, this would be expressed by a colored, circular background on the uniform insignia. The basic uniform became a long- sleeved tunic of gray or tan, with foot coverings built into the uniform pants.

(As you can see, the chronology here is way off, written as it was prior to TNG and the establishment of TNG's Season 1 in 2264 as per TNG: "The Neutral Zone". Mr Scott's Guide was based on the FASA Star Trek RPG timeline, which was in turn derived from the Star Trek Spaceflight Chronology.)

To be fair, the whole thing stemmed from a production mistake in TOS: "The Omega Glory". The article I linked to is pretty interesting and explains how this happened. It was only until the memo surfaced that we discovered exactly what the original intent was.