If you're okay with some kit building, the RC2014 is relatively inexpensive and can be built with all new chips (except graphics, which I can't find a solution I like for that doesn't use a retro chip).

But it's small, easy to understand, and can store and load data from CF cards. Ships with BASIC in ROM, but there are other environments available, like at least one FORTH interpreter.