https://chat.qwen.ai/

https://wan.video/

https://wan.video/

Hello,

I have been looking into a new laptop, and was coming across ones with these NPUs heavily advertised in them. Doing some reading, they don't seem extremely functional at this stage.

They are around 45-50 TOPS at the highest it seems. I found some articles and comments suggesting that 'could' be useful for locally using smaller models, but also statements conflicting with that. As well, most, if not all, 'technical use' of them seem locked into the Windows environment. Even some program from AMD allowing local LLM use requires a Windows Server for it to communicate with, iirc. (AMD, GAIA)

So, is there, currently, any technical use for these, such that it makes much sense to grab a device with one for tinkering?

I'd considered experimenting with smaller models and seeing what comes of those (if small model improvements come through as DeepSeek proponents might suggest).

I'm also just generally new to the technology, but intrigued by the potential to localize usage; not only because of the potential to limit the environmental impact of large data center use.

Any comments, ideas, suggestions, or general pointing in a direction is very appreciated.

Thank you for taking the time. Have a good day!

Hallucinations are destroying under-resourced languages

These have been abundant even before #GenAI, when they were generated by machine translation. And for whatever motivation naive users have flooded crowdsourced resources with such hallucinations.

https://www.technologyreview.com/2025/09/25/1124005/ai-wikipedia-vulnerable-languages-doom-spiral/

@llm

Yesterday I had a brilliant idea: why not parse the wiki of my favorite table top roleplaying game into yaml via an llm? I had tried the same with beautfifulsoup a couple of years ago, but the page is very inconsistent which makes it quite difficult to parse using traditional methods.

• https://dsa.ulisses-regelwiki.de/Kul_Auelfen.html
• https://dsa.ulisses-regelwiki.de/erw_zauber_sf.html?erw_zaubersf=Alchimieanalytiker
• https://dsa.ulisses-regelwiki.de/KSF_Alter_Adersin.html

However, my attempts where not very successful to parse with a local mistral model (the one you get with ollama pull mistral) as it first insisted on writing more than just the yaml code and later had troubles with more complex pages like https://dsa.ulisses-regelwiki.de/zauber.html?zauber=Abvenenum So I thought I had to give it some examples in the system prompts, but while one example helped a little, when I included more, it sometimes started to just return an example from the ones I gave to it via system prompt.

To give some idea: the bold stuff should be keys in the yaml structure, the part that follows the value. Sometimes values need to be parsed a bit more like separating pages from book names - I would give examples for all that.

Any idea what model to use for that or how to improve results?

Since openai removed access to 4.5, I am looking for something comparable from any other company.

Personally, I used it when 4o was not good enough. 4.5 was way better at research and doing more complex programming tasks.

What is comparably good in your experience?

They cry when companies profit from their work, while ignoring the most blatant solution from the start: the AGPL.

Now, its libre software license text file has been replaced with a fake, banning us users from freely forking new versions.

Open WebUI v0.6.6+ ... now adds a ... branding ... clause.

The original BSD-3 license continues to apply for all contributions made to the codebase up to and including release v0.6.5.

Open WebUI lets you download and run large language models (LLMs) on your device using Ollama.

Install Ollama

See this guide: https://lemmy.world/post/27013201

Install Docker (recommended Open WebUI installation method)

1. Open Console, type the following command and press return. This may ask for your password but not show you typing it.

sudo pacman -S docker

2. Enable the Docker service [on-device and runs in the background] to start with your device and start it now.

sudo systemctl enable --now docker

3. Allow your current user to use Docker.

sudo usermod -aG docker $(whoami)

4. Log out and log in again, for the previous command to take effect.

Install Open WebUI on Docker

1. Check whether your device has an NVIDIA GPU.
2. Use only one of the following commands.

Your device has an NVIDIA GPU:

docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda

Your device has no NVIDIA GPU:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Configure Ollama access

1. Edit the Ollama service file. This uses the text editor set in the $SYSTEMD_EDITOR environment variable.

sudo systemctl edit ollama.service

2. Add the following, save and exit.

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

3. Restart the Ollama service.

sudo systemctl restart ollama

Get automatic updates for Open WebUI (not models, Ollama or Docker)

1. Create a new service file to get updates using Watchtower once everytime Docker starts.

sudoedit /etc/systemd/system/watchtower-open-webui.service

2. Add the following, save and exit.

[Unit]
Description=Watchtower Open WebUI
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/bin/docker run --rm --volume /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once open-webui
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

3. Enable this new service to start with your device and start it now.

sudo systemctl enable --now watchtower-open-webui

4. (Optional) Get updates at regular intervals after Docker has started.

docker run --rm --volume /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once open-webui

Use Open WebUI

1. Open localhost:3000 in a web browser.
2. Create an on-device Open WebUI account as shown.

I'm running ollama with llama3.2:1b smollm, all-minilm, moondream, and more. I am able to integrate it with coder/code-server, vscode, vscodium, page assist, cli, and also created a discord ai user.

I'm an infrastructure and automation guy, not a developer so much. Although my field is technically devops.

Now, I hear that some llms have "tools." How do I use them? How do I find a list of tools for a model?

I don't think I can simply prompt "Hi llama3.2, list your tools." Is this part of prompt engineering?

What, do you take a model and retrain it or something?

Anybody able to point me in the right direction?

Did any of you already took a look at the A2A protocol page on github?

cross-posted from: https://lemmy.dbzer0.com/post/41844010

The problem is simple: consumer motherboards don't have that many PCIe slots, and consumer CPUs don't have enough lanes to run 3+ GPUs at full PCIe gen 3 or gen 4 speeds.

My idea was to buy 3-4 computers for cheap, slot a GPU into each of them and use 4 of them in tandem. I imagine this will require some sort of agent running on each node which will be connected through a 10Gbe network. I can get a 10Gbe network running for this project.

Does Ollama or any other local AI project support this? Getting a server motherboard with CPU is going to get expensive very quickly, but this would be a great alternative.

Thanks

This is an update to a previous post found at https://lemmy.world/post/27013201

Ollama uses the AMD ROCm library which works well with many AMD GPUs not listed as compatible by forcing an LLVM target.

The original Ollama documentation is wrong as the following can not be set for individual GPUs, only all or none, as shown at github.com/ollama/ollama/issues/8473

AMD GPU issue fix

1. Check your GPU is not already listed as compatibility at github.com/ollama/ollama/blob/main/docs/gpu.md#linux-support
2. Edit the Ollama service file. This uses the text editor set in the $SYSTEMD_EDITOR environment variable.

sudo systemctl edit ollama.service

3. Add the following, save and exit. You can try different versions as shown at github.com/ollama/ollama/blob/main/docs/gpu.md#overrides-on-linux

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

4. Restart the Ollama service.

sudo systemctl restart ollama

Ollama lets you download and run large language models (LLMs) on your device.

Install Ollama on Arch Linux

1. Check whether your device has an AMD GPU, NVIDIA GPU, or no GPU. A GPU is recommended but not required.
2. Open Console, type only one of the following commands and press return. This may ask for your password but not show you typing it.

sudo pacman -S ollama-rocm    # for AMD GPU
sudo pacman -S ollama-cuda    # for NVIDIA GPU
sudo pacman -S ollama         # for no GPU (for CPU)

3. Enable the Ollama service [on-device and runs in the background] to start with your device and start it now.

sudo systemctl enable --now ollama

Test Ollama alone

1. Open localhost:11434 in a web browser and you should see Ollama is running. This shows Ollama is installed and its service is running.
2. Run ollama run deepseek-r1 in a console and ollama ps in another, to download and run the DeepSeek R1 model while seeing whether Ollama is using your slow CPU or fast GPU.

AMD GPU issue fix

https://lemmy.world/post/27088416

Use with Open WebUI

See this guide: https://lemmy.world/post/28493612

y2u.be/aVvkUuskmLY

Llama 3.1 (405b) seems 👍. It and Claude 3.5 sonnet are my go-to large language models. I use chat.lmsys.org. Openai may be scrambling now to release Chatgpt 5?

I'm an avid Marques fan, but for me, he didn't have to make that vid. It was just a set of comparisons. No new info. No interesting discussion. Instead he should've just shared that Wired podcast episode on his X.

I wonder if Apple is making their own large language model (llm) and it'll be released this year or next year. Or are they still musing re the cost-benefit analysis? If they think that an Apple llm won't earn that much profit, they may not make 1.

Click Here to be Taken to the Megathread!

from !fosai@lemmy.world

Vicuna v1.5 Has Been Released!

Shoutout to GissaMittJobb@lemmy.ml for catching this in an earlier post.

Given Vicuna was a widely appreciated member of the original Llama series, it'll be exciting to see this model evolve and adapt with fresh datasets and new training and fine-tuning approaches.

Feel free using this megathread to chat about Vicuna and any of your experiences with Vicuna v1.5!

Starting off with Vicuna v1.5

TheBloke is already sharing models!

Vicuna v1.5 GPTQ

7B

• Vicuna-7B-v1.5-GPTQ
• Vicuna-7B-v1.5-16K-GPTQ

13B

• Vicuna-13B-v1.5-GPTQ

Vicuna Model Card

Model Details

Vicuna is a chat assistant fine-tuned from Llama 2 on user-shared conversations collected from ShareGPT.

Developed by: LMSYS

• Model type: An auto-regressive language model based on the transformer architecture
• License: Llama 2 Community License Agreement
• Finetuned from model: Llama 2

Model Sources

• Repository: https://github.com/lm-sys/FastChat
• Blog: https://lmsys.org/blog/2023-03-30-vicuna/
• Paper: https://arxiv.org/abs/2306.05685
• Demo: https://chat.lmsys.org/

Uses

The primary use of Vicuna is for research on large language models and chatbots. The target userbase includes researchers and hobbyists interested in natural language processing, machine learning, and artificial intelligence.

How to Get Started with the Model

• Command line interface: https://github.com/lm-sys/FastChat#vicuna-weights
• APIs (OpenAI API, Huggingface API): https://github.com/lm-sys/FastChat/tree/main#api

Training Details

Vicuna v1.5 is fine-tuned from Llama 2 using supervised instruction. The model was trained on approximately 125K conversations from ShareGPT.com.

For additional details, please refer to the "Training Details of Vicuna Models" section in the appendix of the linked paper.

Evaluation Results

Vicuna is evaluated using standard benchmarks, human preferences, and LLM-as-a-judge. For more detailed results, please refer to the paper and leaderboard.