58
Uses for local AI?
(lemmy.mtate.me.uk)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 24 Jul 2024
58 points (100.0% liked)
Selfhosted
39905 readers
294 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
Any model recommendation for that?
The ones i tried get stuck in a loop at some point due to the small context windows.
the answer is very spesific to ur pc and amount of vram you have availşble to you. But anything lama 3 even 8b models finetuned to DM or write stories should theoritically work. The other reply that reccomends connecting to another program to make sure rules are consistent sounds like a great idea whşch I have not tried. I use silly tavern as the ui whşch has lots of options and shit to mske thşngs wkrk well. I would reccomend goşng şnto the "KoboldAI" discord and askşng şn the support sectşon folk there are very helpfull sorry for not beşng able to gşve a strsight answer Also boost the context size way up that shit makes dşfference I habe like 16k or sumthin. good luck!
What on earth is going on with your keyboad?!
Besides that, i have 20GB of VRAM and 64GB or RAM. I can run the mixtral 8x7b model relatively usable. Currently i use oobabooga the most.
I type very poorly on my phone. with that much vram ypu csn get somethşng lşke a 70b model defineyly ask around in the koboldai community that shşt's crszy
Yeah even gpt4o couldn't keep track of encounters, run battles etc. in my case...
I think if you wanted to do it mechanically consistently you'd probably need to integrate it into a vtt where you give it context and potentially fine-tune it to give quest related summaries & gming rather than just "stuff"
I don’t know how tech savvy you are, but I’m assuming since your on lemmy it’s pretty good :)
The way we’ve solved this sort of problem in the office is by using the LLM’s JSON response, and a prompt that essentially keeps a set of JSON objects alongside the actual chat response.
In the DND example, this would be a set character sheets that get returned every response but only changed when the narrative changes them. More expensive, and needing a larger context window, but reasonably effective.
VTT integration would be one hell of a job to do.