12
LLM for GTX1080 (8GB) for local use. (lemmy.dbzer0.com)
submitted 1 year ago by pythia@lemmy.dbzer0.com to c/oobabooga@lemmy.world
7 comments fedilink hide all child comments

Could someone recommend a LLM for the Nvidia GTX1080? I've used the gptq_model-4bit-128g of Luna AI from the Bloke and i get a response every 30s-60s and only 4-5 prompts before it starts to repeat or hallucinate.

you are viewing a single comment's thread
view the rest of the comments
[-] SkySyrup@sh.itjust.works 3 points 1 year ago

yeah that should work!

[-] pythia@lemmy.dbzer0.com 2 points 1 year ago

Yes it does and fits the GPU just fine. Didn't hallucinate but it was slow like 60s+ in the first run but did it's job. Thanks.

[-] SkySyrup@sh.itjust.works 2 points 1 year ago

good to hear it worked, it’s weird it’s so slow. I’m lucky to have access to a 3060, which isn’t that far out from a 1080, and get at least 40t/s on it. Are you running on CPU or are you using exllama?

[-] pythia@lemmy.dbzer0.com 1 points 1 year ago

It's running on gpu, the task-manager shows 92% GPU utilization and i chose exllamaV2.

[-] SkySyrup@sh.itjust.works 2 points 1 year ago

that’s really weird, I’m not sure how to help you there unfortunately :(

this post was submitted on 26 Oct 2023
12 points (100.0% liked)

Oobabooga Text Generation

2 readers
1 users here now

Community for Oobabooga / Pygmalion / TavernAI / AI text generation

Let's rebuild our knowledge base here!

Ooba community is still dark on reddit, so we're starting from scratch. https://www.reddit.com/r/Oobabooga/

Subscribe, engage, post, comment!

Helpful links:

https://github.com/oobabooga/text-generation-webui

https://zoltanai.github.io/character-editor/

https://www.chub.ai/characters

Remember to mark NSFW posts.

Other AI communities to check out:

!fosai@lemmy.world

founded 2 years ago
MODERATORS
2dollarsim@lemmy.world