Has anyone here really pushed the multimodal stuff with this yet? Been running this on a 4090 for over 24 hours and have been meaning to try out if its able to summarize images well, especially compared to something like qwen3vl on the same hardware
Anybody tried it on MLX? I was really skeptical about them releasing 27b, after they skipped it for 3.7 but I'm pleasantly surprised.
Cheers for the new local GOAT!
Also on ModelScope in case you-know-who bribes huggingface takedown because model is too dangerous.
I’m not sure why there’s so much focus on comparing Qwen’s 27B models with older Opus versions. 3.6 was pretty comparable to Sonnet 4.6 for coding, and now 3.8 gets it to around Sonnet 5 level (SWE Bench Pro 61.7 vs. S5’s 63.2). That’s pretty a awesome achievement and I think comparing it to Sonnet sets more realistic expectations.
So far, I’ve had 3.8 do a code review and also review some creative writing. It’s quite a bit more thorough than 3.6.
Qwen3.6 with 4-bit quant was hallucinating HARD. When asked to summarize the plot of obscure, cheesy 80s books, it confidently made up plot and characters. Not a single statement was correct. It was glorious.
Any higher quant and it would take 30-90 seconds before first token. Both MLX and GGUF on an M3Max with 128G combined RAM. Will have to try it with this one.
In my testing so far, it seems to lean more into "I don't know" type answers, but I haven't poked at it that much yet.
One of the queries that used to consistently trip up Qwen 3.5/3.6 was asking directly about a plausible sounding (but non-existent) work (e.g. Are you familiar with "A Valiant Effort (1989)"?); stock Qwen 3.8 will say it doesn't know or isn't sure and ask me for context.
27b fits easily into 128 unified RAM, even unquantized. So something was definitely wrong.
Running 3.8-27b on my Strix Halo box right now.
Hey, I also have a Strix Halo box. Which model are you running exactly? The Q8 and BF16 are half the size of Qwen3-Coder-Next-UD-Q8 I have been running before. I tested 3.8-BF16 and UD-Q8 but both run at abysmal speed in comparison to Coder Next (around 8 t/s during chat). Which one gives you best results?
You need to use MTP (Multi token prediction) to get acceptable speeds on a Strix Halo. On llama.cpp you just need to add --spec-type draft-mtp --spec-draft-n-max 3 -np 1 to your arguments.
You should see around 15-20 t/s in coding tasks at Q8, which will still be much slower than Qwen3-Coder-Next-UD-Q8 since it's 27b active parameters instead of 3b.
As for Q8 vs BF16, it usually makes barely to no difference in quality. Speed is higher on Q8 though. Still testing with 3.8 since it just released.
Thank you, I will check this out. So what I got from this is that the speed may be slower but results should be more accurate?
Yes, 27b is going to produce much better results.
On a DGX with 128G unified I could push Qwen3.6-27B to 8-bit and it had pretty decent response. It just croaked with MLX on the Mac.
Waiting to see what Apple announces next month. Judging by RAM prices, it'll be waaay outside my budget.
It did a good job with coding, even with the quantization. Did they prioritize that in its training perhaps, at the expense of other things?
Goddamn, number go up in many benches.
Having issues with it (Unsloth's gguf).
0.27.136.404 I srv proxy_reques: proxying request to model unsloth/Qwen3.8-27B-GGUF:Q4_K_M on port 55741
[55741] 0.14.424.772 W srv operator(): got exception: {"error":{"code":500,"message":"\n------------\nWhile executing CallExpression at line 106, column 32 in source:\n...first %}↵ {{- raise_exception('System message must be at the beginnin...\n ^\nError: Jinja Exception: System message must be at the beginning.","type":"server_error"}}
Got an answer - there were indeed issue with some quants
Actually edit - UD-Q4_K_XL and all UD-* has our corrected chat teaplate
We need to update the non UD-* ones with our chat template - stay tuned
All quants updated with correct template now
Waiting for 2 or 3 such iteration before testing it on my gear, but thanks for your services, brave frontier soldier o7
Sounds like you could just replace the template to fix it, there's a popular Qwen fixed template on hugging face, try that
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/blob/main/chat_template.jinja
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.