It also demonstrates how much AI companies mislead the public on what their products can do. If a guy is selling lawnmowers that actually just generate grass clippings without mowing the lawn, you’re not an idiot for thinking it was going to mow grass.

I mean, you literally have whole videos on YouTube made by GothamChess who shows how LLMs play chess. They literally spawn pieces from air, play moves that are illegal etc.

Using an LLM to play chess is like using autocorrect to write a novel.

And that's the big problem with AI right now. People don't understand what it is, they just want the label slapped on to as many things as possible.

AI is the new IoT, it will be integrated into everything, less than useless for 99.9% of consumers, and yet, still wildly successful.

Hundreds of billions of dollars spent

No profitable product

No consistently usable product other than beginner code tasks

Massive environmental harms

Tens of thousands of (useful!) careers terminated

Destroyed Internet search, arguably the one necessary service on the Internet

No chance it's going to get better

Atari 2600 beating it at chess is a perfect metaphor. People who want to complain about it can bite its plastic woodgrain printed ass.

I'm quite sure that the guy understood pretty well what LLMs can do. He just wanted to deinflate all the bullshit promises by Techbros

I bet the llm doesn’t even know what en passant is

By my best feelings, this shit is a bigger bust than the .com bubble, and I predate that latter shit by roughly twenty years.

LLM sucks at maths, sucks at chess, sucks at remembering stuff and being consistent ... They suck at everything a computer is usually good at.

I'm quite sure that you could use a LLM to play chess and probably even successful, but you need to train it on chess notation of games instead of a pile of fanfiction and other copyright infringements. I have considered trying that but was turned off by how inaccessible LLM training is and how difficult it would be to get a sufficient amount of games written in proper chess notation. Obviously this would not be a real LLM, as it does not "speak", but I was curious how well this would work utilizing the same technique.

Okay, i think there is quite a misunderstanding here.

Some older versions of LLMs (chatgpt3.5-turbo-instruct) can play chess relatively well (around 1750 Elo) : here is a link to an article studying that.

Some points :

• it is of course way worse than almost any algorithm designed for chess
• one of the reason we cannot get these result back (at least not that good, here is a link to a blog post of someone making recent LLMs chatbots better at chess) could be that we do not have access to pure completion mode on models trained on selected data (where they could purposefully choose only good chess matches), and those are now hidden behind a chatbot layer instead.
• it seems to reveal that models have a somehow accurate representation of the chess board when predicting chess moves
• it seems to have a quite unique feat that is : if you feed them a prompt that say they play as a very good player, and then the beginning of a game with a blatant bad move (giving away a queen for example), they sometimes play the entire game with moves that purposefully give away pieces, as if they guess that the only reason they would lose a piece that easily is by purposefully losing them. It has close to zero utility, but it's interesting anyway.

Asking ChatGPT to play chess is like asking someone who's not played Chess to play well, and then documenting how poorly it played. Like no shit the hammer did a bad job as a saw. You wanted it cut, you should have used the tool for the job.

ChatGPT isn't Deep Blue. It's not made for that. You're asking a word processor to calculate pi.

ChatGPT hallucinates moves so it'll almost always lose by the arbiter coming second time because of illegal move.

Nobody thought it would do very well. This was a software dev's little diversion.

We should praise attempts to make the public aware of the limitations of LLMs, not laugh at the guy who did this.

That's a very good move! To counter, you should follow these three principles:

• Prepare a response move that will prevent a future good move.

• Defend your own pieces and try to attack theirs.

• Don't be too eager to sacrifice pieces in order to make short term gains.

• Be prepared to sacrifice an unimportant piece to make a good gain.

If you want to make a good move, try Rook H8 -> G7.

You could train an AI just to play chess. Sites like chess.com have tens, hundreds of millions of games to use as training data. But the AI isn't "thinking" though, it's just being asked given this input, what's the most likely outputs and picking one based on its settings. Then the other player moves, the context updates, rinse, repeat. Such an AI would likely whoop most people's asses but experienced players might figure how to lead it down a path where it doesn't sufficient training data to play strongly.

But it's not a generalized LLM like ChatGPT where it's picking up a handful of chess games from god knows without knowing or enforcing the rules or anything else.

Likewise I bet we'll see AIs for poker and other lucrative online sports. I bet a lot of online casinos have amassed huge stores of data to produce AIs, as well as players using scraping or logs to do the same. I could even see online casinos running AIs in games because it's a way of taking money from players beyond the normal rake.