102
Which one of them made these overlapping bar labels?
Hey, a lot of people sucked making matplotlib plots before LLMs existed.
Probably alibaba with Qwen
This seems like a great time to mention what I built and hosted months ago. https://masland.tech/ai-efficiency-index/
The AI Efficiency Index tracks the cost/intelligence mix. Low intelligence is useless. Capability scales exponentially. Expensive ≠ better value. MiMo-V2.5 is the leader.
If you have any questions feel free to ask. This uses the Artificial Analysis leaderboard. They have "cost per task" which I think is weird.
It’s a bit misleading.
Qwen 27B has way less “world knowledge” than GPT-5. Ask it random trivia without internet search access, and GPT would know waaay more.
This is generally true of small vs large models.
…But honestly, Qwen 27B is better at tool use or agentic stuff. It’s hyper optimized for just that and coding assistance, basically.
This is often true of old vs new. Most newer models have hyper focused on agents/coding, often to the detriment of other use cases.
Quantization for practically running Qwen 27B also has an impact. A off-the-shelf Q4_K_M is not the same as the unquantized weights in real-world use, or even an “optimized” quantization like a custom exl3.
I mean baking knowledge into a model isn't really all that useful to begin with. Just download wikipedia locally and have it access it through tool use, it's way more efficient and more accurate. And yeah, I find Q6 tends to be the sweet spot where it's close enough to full 16 bit in performance, but doesn't chew up too much memory.
This is a pretty good idea I hadn't thought about. The whole wikipedia is about 37Gb.
yeah, it's not a completely insane amount of data, and a db like postgres can do fast text search on that too with fuzzy matching
What version do you use and how do you run Qwen3.6? I've played around a bit with the Q4 version in LM studio +Zed, but I was not happy with the results. It looses track very often and often enters infinite loops or just stops...
I use an exl3, with 4 bit MLPs but higher bit depth attention layers. And I force some custom sampling so I can lower the temperature a bit while keeping it out of loops.
This won’t work in LM Studio though. You have to run such a thing in TabbyAPI or some other backend that supports exllamav3.
Ok, thanks! Thaf sounds quite advanced, but I'll have a read afterwards :)
That's why you couple it with your own, self-hosted yacy or searxng instance. Embedded world knowledge does not help a model if it becomes outdated. I just let my agent research, embed that knowledge to a little Qdrant server, so other servers are not bothered again and pull the information from there when needed again. With a little RAG you can have GPT at home.
If you run it with RAG (retrieval augmented generation), where the model includes web search, you get pretty decent results. I often include the phrase "search the web extensively for up to date information" in my prompts.
I sort of get it because of you know you're only using the model when you have Internet and want it to search then just let it.
🙄
Is it still telling people to put glue on their pizza and convincing teenagers to kill themselves?
what kinda hardware do you need to "run this on your desktop"?
A P40 with 24GB is ~150€, a V100 (32GB) is ~600€. Both of these fit Qwen3.6 27B (The P40 is about 3x slower though). The V100 even fits 400k context with a Q4 KV-Cache , which means you can have two slots for parallel processing (llama-cpp). You don't even have to use system memory. One of my inference servers is running with 8gb of DDR3 and a 2nd Gen i7, so my old hardware has a good use again.
thanks, good stuff! sadly, got none of these locally available.
I had to get mine from ebay and wait a couple weeks as they came from China too. Also I recommend having a 3D-Printer and some Blower-Fans on hand as you will either have to buy or print your own fan shroud for these server cards.
You need a GPU with around 16gb vram at a minimum to run qunatized version.
load more comments
(2 replies)
load more comments
(3 replies)
according to performance on standard benchmark. somewhat covered by the controversy surrounding the term: benchmaxing.
if you see all benefit as a linear one dimensional height on a bar graph..
its almost like you assume that the previous model gave the same exact answer(same style) and the new mode gave the same exact answer PLUS additional useful information.
it might be convenient if measuring progress was so simple. but unfortunately/fortunately , it is not so simple .
the most important benchmark are the comparison of outcomes on the problems that YOU have & prompts that YOU can(will) write. nothing else matters for YOU.
- i admit benchmarks are well designed to objectively measure competence on challenging problems that require skill and really need only ONE correct answer.
Sure, a benchmark doesn't capture all the subtleties and different use cases, but it does give a general idea of the capabilities of a model. Obviously, you have to run the model and see if it does what you need. But the chart isn't really about the nuance, it's showing how drastically the efficiency of the models has improved in just a year. The fact that we can even reasonably compare a model you can run on a desktop to one that needed a data center just a year ago is phenomenal.
i see what your saying. i didnt mean to discredit standard benchmarks entirely.
i guess its obvious that it measures capability regardless of imprecision.
2 major proposed changes:
**first, i dont really know. aside from saying "benchmark your own prompt+usecase"
a proposed plan:
- approach one: pay attention and credit new or improved architecture designs and research.
- approach two: spend more attention on benchmarks. especially specific benchmarks ( that are not focused with industrial domain tasks.) **domain task pursuit, is useful!.. but it depends on if your interest align to popular domains.
- approach three: if willing to utilize remotely hosted models. rating should also take in consideration.. tools and everything else: websearch performance, RAG performance, smooth interface, pref/balance between speed vs comprehensiveness, cost (if relevant), etc.. .
Honestly, I think the most reasonable approach is just to see what other people's experience is like and which models are well regarded, then try them out and see which one is the best fit for what you're doing. You might not even need the top performing one necessarily, and speed or lower resource usage might be a bigger factor.
Those models are for general use. If you have business use case and data related to it you can finetune model for specific use that will outperform all of frontier models and run at fraction of cost.
that's the other huge advantage of open models you can run locally
How can you run a local model for better coding support than fable/opus?
You can't right now because there is no dataset available for fine tuning. I'm just saying that it's possible to fine tune 8B-35B parameter model in int4 that will outperform those models ex. for single programming language and developer specific problems. You can read example of fine tuned 8B model here https://fermisense.com/when-machines-take-the-wheel/
Ok, but all of them are far too inefficient for what they deliver.
Ok, but that's a completely nonsensical statement. If you ever used Qwen in an agentic loop, you'd know that it delivers working code, and it takes about same resources as playing a modern game, and I don't see anybody whinging that game are too inefficient for what they deliver.
Sure they are. Especially with moores law dead. And especially since 2020; the more million dollars behind it, the more they run like one from a inexperienced single-dev on itch.
Not sure what Moore's law has to do with anything here to be honest. The models you can run locally on a consumer desktop can do real work, and their resource usage is no different from any other software like games that you'd run.
_
Gotta wait till Qwen 4 for proper font scaling.
Gotta wait till Qwen 4 for proper font scaling.
Kimi is better. Waiting for it to appear on Ollama Cloud.
The difference is that you can run Qwen completely local though.
So which one tried to cheat its way into the test?
Llama is one for sure
view more: next ›
this post was submitted on 28 Jul 2026
102 points (100.0% liked)
Technology
43033 readers
100 users here now
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
founded 7 years ago
MODERATORS