13
LLM with Web Search functionality
(lemmy.ml)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 21 Jun 2026
13 points (100.0% liked)
Self Hosted - Self-hosting your services.
20016 readers
4 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules
- No harassment
- crossposts from c/Open Source & c/docker & related may be allowed, depending on context
- Video Promoting is allowed if is within the topic.
- No spamming.
- Stay friendly.
- Follow the lemmy.ml instance rules.
- Tag your post. (Read under)
Important
- Lemmy doesn't have tags yet, so mark it with [Question], [Help], [Project], [Other], [Promoting] or other you may think is appropriate. This is strongly encouraged!
Cross-posting
- !everything_git@lemmy.ml is allowed!
- !docker@lemmy.ml is allowed!
- !portainer@lemmy.ml is allowed!
- !fediverse@lemmy.ml is allowed if topic has to do with selfhosting.
- !selfhosted@lemmy.ml is allowed!
If you see a rule-breaker please DM the mods!
founded 5 years ago
MODERATORS
Openwebui+searxng on a AMD strix board.
Pro: works like a charm, low power consumption, fast, "big" , LLM (running qwen3.6 35B A3B + gemma4 E4B for website summaries and other smaller tasks)
Con: strix boards start at 2k€, more in USA because of tarrifs
Curious why do you swap between Qwen and E4B. On my hardware they perform with similar tps. Qwen 3.6 35B spits out 80-100tps on AMD 9700 and E4B gives me about the same tps.
To avoid context switching on the GPU. OpenWebUi for example uses it for memory and title generation.
Those are not performance critical and background tasks, so instead of slowing down qwen, we just outsource this stuff to the NPU.
Edit: see here for more details
Oh I see. Okay this makes sense. I just throw Qwen 3.6 35B Q8 on 2 GPUs and use it for everything but coding agent.
For those who want to know more, rough setup:
We get 70-100tok/s generation. Four slots with 256k context length each.
We use a smaller Board with "only" 64GB of shared LPDDR5X. Bottleneck is memory speed, rocmfp4 quants help a lot.
As soon as I get my imatrix calibration right, I will publish the quantized versions.
Most existing quantized models are broken. The authors did some not supported stuff (like using a already quantized model and requantize it) that you may get issues with coherence or sudden Chinese words in the output.
That is not an issue with rocmfp4 but with vibe coders and agent psychosis.
Do you have a walk through for setup?
I'm on the strix halo 128 gb variant and while I got ollama working fine, i haven't gotten any of these multi headed setups working
I am on Gentoo for it, but everything with a decent rocm should work.
Have a look for llama-swap, that handles multi head endpoints.
Also, as you are on a big board, you can quantize yourself, as the BF16 version of qwen has only 72gb.
I will try and post a full writeup next days. But feel free to dm me, if you need some guidance on quantize or more.
I am using this fork currently: https://github.com/charlie12345/ROCmFPX
Stuff happens fast currently, so may be worth to wait a week or two ig you need something super stable, but if you are up for experimenting, that's the way to go
THis is great, thanks. I'm on the z-13 and needed to use it for a work project, which is wrapping up soon. I'm planning on re-building it as a locally hosted agent support machine.
Great man! Gentoo lover and long time addicted here.... Keep it the good work!
Thank you so so much for pointing out ROCmFP4. I have been tinkering with my RDNA 3 framework on llama. I was struggling with ROCm llama.cpp and have been using vulcan in the meantime. I know there’s some issues on the llama.cpp github to try and fix my issue (UMA stuff), but haven’t come across this specific project. Gonna try it out
Yup. And if you want to take a small step without major hardware requirements: connect your setup to a paid subscription Mistral or Anthropic API. They allow you to switch off training on your data.
On top of that, the costs are way lower than the normal consumer grade chat subscriptions, and your searches + memory are kept locally (e.g., managed through open webui).
Openrouter is also nice for this. You can use real cheap models for embedding and the bigger ones for the actual research.
What's a "strix board"? Is that necessary?
AMD Strix is an APU, optimized for AI. It is the cheapest option I am aware of to run bigger models at home. 2k for 56GB VRAM, and less den 300W total power Budget.
One could run smaller models. But for the context sizes required for research work, that is nearly impossible.
Also, external services, like openrouter, can be used to use models hosted in the cloud.
But for self hosted, you need something that can run models with at least 15GB of VRAM + Context. For comparison. Our highly quantized model uses 20GB of vram. For our 4 slots we need another 20GB on top of it (around 5GB for 254k tokens), making it 40GB.