these smaller local models tool a massively smaller footprint to train, are much less restrictive and got a generally different vibe to them.
they are dumber than what big evil tech is selling to us, but... they are locally running on your lappytoppy or desktop.

for an easy gateway, pull ollama and run ollama run qwen3.5:4b. that commands pulls and runs a pretty okay Qwen model on your GPU or CPU (whatever ollama finds on your machine).
(qwen is a chinese model, by alibaba, so... if you don't like china, maybe run gemma4:e4b instead. it's by google...)

if you want more of those "literally just a text predictor" vibes, i hiiiiighly recommend llama-cpp. it's faster, it's good, it's what everything else is built on AND it even has a nice ui.
you can pull the release here (i recommend the "vulkan" one, even if you only have an iGPU).
once unpacked, pull some "gguf" model that fits in your VRAM or RAM from somewhere. if you have about 15 GB of RAM free on your system, you may want to try Qwen3.6-35B-A3B or it's no-conversation-training counterpart Qwen3.5-35B-A3B-Base at some lower quantization like the "IQ2_M" one.

• click on the quant
• click the "download" that appears in the sidebar
• wait for it to finish

finally, run it with the "llama-server" binary you find in the llama cpp folder like so:

# ctk and ctv make it go faster
./llama-server --model ~/.llama-cpp/models/qwen3-coder/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf -ctk q4_0 -ctv q4_0 --port 9090
0.00.036.875 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.037.353 W srv  llama_server: -----------------
0.00.037.356 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.037.356 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.037.356 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.037.356 W srv  llama_server: -----------------
0.00.038.704 I srv    load_model: loading model '/home/maria/.llama-cpp/models/qwen3-coder/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf'
0.00.641.956 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.12.362.730 I cmn          init: llama threadpool init, n_threads = 2
0.13.447.376 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 101632, kv_unified = 'true'
0.13.474.204 I srv  llama_server: model loaded
0.13.474.220 I srv  llama_server: listening on http://127.0.0.1:9090/

open that page http://127.0.0.1:9090/ and see what goes.

ollama automatically creates a server in the background which you can curl from like this:

curl 127.0.0.1:11434/v1/chat/completions -d '{
> "messages": [
>   { "role": "user", "content": "heyhiiii ~ ~ ~ ~ ~ how we doinnnn?" }
> ]
> }'

llama-cpp is the same, but you gotta start the server like i showed earlier. these are standard "OpenAI-compatible" APIs. meaning: you can use them in agents and such if you wanna.

buuuuuut yeaaaa it kinda still is very much... LM stuff. so i can definitely understand staying far away from this.