Generally in trees you have the xylem in the middle which transports water and minerals from the roots to the rest of the plant. You have the phloem on the outside, it transports photosynthetic products and nutrients to all parts of the plant. The cambium where growth happens sits between them, because there they get easy access to water minerals and nutrients. The xylem is a mix of living and dead cells. The tracheids and vessel elements in the xylem, which are responsible for water transport, are dead at maturity. So it's probably too hard to move them to grow from the inside out for trees. This whole process of growing thicker is called secondary growth as opposed to primary growth at the tips of stems and roots.

Also if were the other way around and trees would grow from the inside out, you'd have to have vessels going from the leaves to the center of the stems to deliver nutrients, which just complicates everything. And the tree wanted to grow from the inside by 1cm, every "ring" in the xylem would have to grow one centimeter longer in circumference or crack. It's much easier to just add a layer on the outside. Also having living layer around the tree probably helps it defend itself from pathogens. If all the mostly dead woody stuff was on the outside fungi etc would have an easier time invading, I think.

Trees grow by adding layers to what they already laid down in the past. The newest layer is indeed on the outside.

well i'd say that trees grow in both ways: the stem grows on the outside because it's acting more like our bones, but the bark acts like our skin and indeed grows in the same way, new matter being created at the base and pushed outwards, hence why a lot of bark has tons of cracks in them from the stretching.

Trees grow layers on the outside, just underneath the bark. This is the only part of a tree trunk that is actively growing and is part of the plant's circulatory system. All the older wood toward the center of the tree is basically only structural support.

There's a technique called "collaring" used to remove invasive trees, if you cut a shallow, 4 inch tall ring from the bark around eye level, everything above that ring will die, because you've severed the circulatory system between the roots and everything above the collar

Does your skin grow from the outside? Does it even make any sense that it would? Why would trees be different?

So, the inner rings are actually the oldest. I'm no botanist, but I know that with your average deciduous tree, you have a layer right between the bark and first wood layers, and that thin layer is where the cell division for growth is mostly occuring. So, it actually is the exterior, just right beneath the bark.

Other kinds of plants are probably different, and again, I'm no botanist. Just have some basic biology study.