Yes, it's a programming thing. If you think about all of the symbols you can see in a website, there are many more than what's available on your keyboard, like the Copyright symbol, ©. Since programmers deal with the same limited set of keyboard keys, everything that can be displayed in text is given a unique number. Lowercase a is 97, the Copyright symbol is 169. Some commonly used ones are also given a slightly more readable name in addition to the number.

Now that there are numbers or names associated with everything, the programmers need a way to tell the browser that this number or text should be used to refer to a symbol. To do this, they use an Escape Character, a symbol that tells the browser to treat what follows as a lookup. That escape character in HTML is the ampersand, so & yen; will display as the symbol for Japanese Yen, ¥. The semicolon denotes the end of the Escape Sequence.

So with Ampersand having a special meaning, to show an actual ampersand you would typically use & amp; and the browser would turn it back into &. However, if you're not looking at the text in a browser, or if the area where the text is displayed doesn't understand the & amp; notation, then you will see it exactly as you described.