49
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 01 Dec 2024
49 points (100.0% liked)
Futurology
1851 readers
47 users here now
founded 1 year ago
MODERATORS
A lot of the smaller LLMs don’t require GPU at all - they run just fine on a normal consumer CPU.
Wouldn't running on a CPU (while possible) make it less energy efficient, though?
It depends. A lot of LLMs are memory-constrained. If you’re constantly thrashing the GPU memory it can be both slower and less efficient.
yeah but 10x slower, at speeds that just don't work for many use cases. When you compare energy consumption per token, there isn't much difference.