Once you've opened them, yeah. Even with refrigeration, they'll go bad within 1½ weeks, sometimes less, so I really don't think you can skip refrigeration, unless you do use it up in like a day or two.

There's oat milk powder you can buy, which you can mix with water to make however much oat milk you need (the unfermented kind). I've only tried one such powder so far, which I really didn't like, but it tasted kind of 'stale', so I'm hoping that's just a defect with that specific product. Other powders do also have different ingredients, so should taste somewhat different anyways.

Well, and what's also possible is to take some oatmeal and the other ingredients I listed above, throw them into a mixer and then strain the remaining oatmeal pulp afterwards. You can eat that oatmeal pulp in cereal or such. You can also throw a ripe banana into the mixer to help make it more creamy, if the banana taste is okay for what you want to use it.