Referencing the telephone game does not prove anything here. The telephone game is shows that humans are not good at copying something exactly without changes, which computers are better at. But the question here is if AI can achieve deeper understanding of a work, which is needed to produce a good summary. This is something humans are far better at. The AI screws up the summary here in ways that no reasonable person who has watched the TV series (or played the games) would ever screw up.

We will secure energy dominance by dumping even more money and resources into a technology that is already straining our power grid. But don't worry. The LLM will figure it all out by reciting the Wikipedia page for Fusion Power.

AI is expected to make cutting-edge simulations run “10,000 to 100,000 times faster.”

Turns out it's not good to assume that literally every word that comes out of a tech billionaire's mouth is true. Now everyone else thinks they can get away with just rattling off numbers where their source is they made it the fuck up. I still remember Elon Musk saying a decade ago that he could make rockets 1,000 times cheaper, and so many people just thought it was going to happen.

We need scientists and engineers. We do not need Silicon Valley billionaire visionary innovator genius whizzes with big ideas who are pushing the frontiers of physics with ChatGPT.

In my experience most people just suck at learning new things, and vastly overestimate the depth of expertise. It doesn't take that long to learn how to do a thing. I have never written a song (without AI assistance) in my life, but I am sure I could learn within a week. I don't know how to draw, but I know I could become adequate for any specific task I am trying to achieve within a week. I have never made a 3D prototype in CAD and then used a 3D printer to print it, but I am sure I could learn within a few days.

This reminds me of another tech bro many years ago who also thought that expertise is overrated, and things really aren't that hard, you know? That belief eventually led him to make a public challenge that he could beat Magnus Carlsen in chess after a month of practice. The WSJ picked up on this, and decided to sponsor an actual match with him and Carlsen. They wrote a fawning article about it, but it did little to stop his enormous public humiliation in the chess community. Here's a reddit thread discussing that incident: https://www.reddit.com/r/HobbyDrama/comments/nb5b1k/chess_one_month_to_beat_magnus_how_an_obsessive/

As a sidenote, I found it really funny that he thought his best strategy was literally to train a neural network and ... memorize all the weights and run inference with mental calculations during the game. Of course, on the day of the match, the strategy was not successful because his algorithm "ran out of time calculating". How are so many techbros not even good at tech? Come on, that's the one thing you're supposed to know!

Just had a conversation about AI where I sent a link to Eddy Burback's ChatGPT Made Me Delusional video. They clarified that no, it's only smart people who are more productive with AI since they can filter out all the bad outputs, and only dumb people would suffer all the negative effects. I don't know what to fucking say.

One of the core beliefs of rationalism is that Intelligence™ is the sole determinant of outcomes, overriding resource imbalances, structural factors, or even just plain old luck. For example, since Elon Musk is so rich, that must be because he is very Intelligent™, despite all of the demonstrably idiotic things he has said over the years. So, even in an artificial scenario like chess, they cannot accept the fact that no amount of Intelligence™ can make up for a large material imbalance between the players.

There was a sneer two years ago about this exact question. I can't blame the rationalists though. The concept of using external sources outside of their bubble is quite unfamiliar to them.

More AI bullshit hype in math. I only saw this just now so this is my hot take. So far, I'm trusting this r/math thread the most as there are some opinions from actual mathematicians: https://www.reddit.com/r/math/comments/1o8xz7t/terence_tao_literature_review_is_the_most/

Context: Paul Erdős was a prolific mathematician who had more of a problem-solving style of math (as opposed to a theory-building style). As you would expect, he proposed over a thousand problems for the math community that he couldn't solve himself, and several hundred of them remain unsolved. With the rise of the internet, someone had the idea to compile and maintain the status of all known Erdős problems in a single website (https://www.erdosproblems.com/). This site is still maintained by this one person, which will be an important fact later.

Terence Tao is a present-day prolific mathematician, and in the past few years, he has really tried to take AI with as much good faith as possible. Recently, some people used AI to search up papers with solutions to some problems listed as unsolved on the Erdős problems website, and Tao points this out as one possible use of AI. (I personally think there should be better algorithms for searching literature. I also think conflating this with general LLM claims and the marketing term of AI is bad-faith argumentation.)

You can see what the reasonable explanation is. Math is such a large field now that no one can keep tabs on all the progress happening at once. The single person maintaining the website missed a few problems that got solved (he didn't see the solutions, and/or the authors never bothered to inform him). But of course, the AI hype machine got going real quick. GPT5 managed to solve 10 unsolved problems in mathematics! (https://xcancel.com/Yuchenj_UW/status/1979422127905476778#m, original is now deleted due to public embarrassment) Turns out GPT5 just searched the web/training data for solutions that have already been found by humans. The math community gets a discussion about how to make literature more accessible, and the rest of the world gets a scary story about how AI is going to be smarter than all of us.

There are a few promising signs that this is getting shut down quickly (even Demis Hassabis, CEO of DeepMind, thought that this hype was blatantly obvious). I hope this is a bigger sign for the AI bubble in general.

EDIT: Turns out it was not some rando spreading the hype, but an employee of OpenAI. He has taken his original claim back, but not without trying to defend what he can by saying AI is still great at literature review. At this point, I am skeptical that this even proves AI is great at that. After all, the issue was that a website maintained by a single person had not updated the status of 10 problems inside a list of over 1000 problems. Do we have any control experiments showing that a conventional literature review would have been much worse?

Lately I've been mildly annoyed when I just want to relax and watch gaming videos on Youtube and I see recommendations for some AI critihype. Out of morbid curiosity, I decided to click on one of them and of course the "original paper" the video is based on is the stupid Anthropic blog post about how the AI blackmailed someone (after it was told to blackmail someone). I was even more annoyed to find out how popular it is, but at least it shows how the general public has such a negative opinion of AI. Some of the comments are thankfully pushing back against the video and focusing on the real harms.

I thought that by now we would have learned from the tobacco companies to never trust "research" done by a company about their own products.

OpenAI claims that their AI can get a gold medal on the International Mathematical Olympiad. The public models still do poorly even after spending hundreds of dollars in computing costs, but we've got a super secret scary internal model! No, you cannot see it, it lives in Canada, but we're gonna release it in a few months, along with GPT5 and Half-Life 3. The solutions are also written in an atrociously unreadable manner, which just shows how our model is so advanced and experimental, and definitely not to let a generous grader give a high score. (It would be real interesting if OpenAI had a tool that could rewrite something with better grammar, hmmm....) I definitely trust OpenAI's major announcements here, they haven't lied about anything involving math before and certainly wouldn't have every incentive in the world to continue lying!

It does feel a little unfortunate that some critics like Gary Marcus are somewhat taking OpenAI's claims at face value, when in my opinion, the entire problem is that nobody can independently verify any of their claims. If a tobacco company released a study about the effects of smoking on lung cancer and neglected to provide any experimental methodology, my main concern would not be the results of that study.

Edit: A really funny observation that I just thought of: in the OpenAI guy's thread, he talks about how former IMO medalists graded the solutions in message #6 (presumably to show that they were graded impartially), but then in message #11 he is proud to have many past IMO participants working at OpenAI. Hope nobody puts two and two together!

Hmm, should I be more worried and outraged about genocides that are happening at this very moment, or some imaginary scifi scenario dreamed up by people who really like drawing charts?

One of the ways the rationalists try to rebut this is through the idiotic dust specks argument. Deep down, they want to smuggle in the argument that their fanciful scenarios are actually far more important than real life issues, because what if their scenarios are just so bad that their weight overcomes the low probability that they occur?

(I don't know much philosophy, so I am curious about philosophical counterarguments to this. Mathematically, I can say that the more they add scifi nonsense to their scenarios, the more that reduces the probability that they occur.)

Username called "The Dao of Bayes". Bayes's theorem is when you pull the probabilities out of your posterior.

知者不言，言者不知。 He who knows (the Dao) does not (care to) speak (about it); he who is (ever ready to) speak about it does not know it.

AI research is going great. Researchers leave instructions in their papers to any LLM giving a review, telling them to only talk about the positives. These instructions are hidden using white text or a very small font. The point is that this exploits any human reviewer who decides to punt their job to ChatGPT.

My personal opinion is that ML research has become an extreme form of the publish or perish game. The most prestigious conference in ML (NeurIPS) accepted a whopping 4497 papers in 2024. But this is still very competitive, considering there were over 17000 submissions that year. The game for most ML researchers is to get as many publications as possible in these prestigious conferences in order to snag a high paying industry job.

Normally, you'd expect the process of reviewing a scientific paper to be careful, with editors assigning papers to people who are the most qualified to review them. However, with ML being such a swollen field, this isn't really practical. Instead, anyone who submits a paper is also required to review other people's submissions. You can imagine the conflicts of interest that can occur (and lazy reviewers who just make ChatGPT do it).

I have a lot to say about Scott, being that I used to read his blog frequently and it affected my worldview. This blog title is funny. It was quite obvious that he at least entertained, if not outright supported, rationalists for a long time.

For me, the final break came when he defended SBF. One of his defenses was that SBF was a nerd, so he couldn't have had bad intentions. I share a lot of background with both SBF and Scott (we all did a lot of math contests in high school), but even I knew that it's not remotely an excuse for stealing billions of dollars.

I feel like a lot of his worldview centers around nerds vs everyone else. There's this archetype of nerds being awkward, but well-intentioned and smart people who can change the world. They know better than everyone else on how to improve the world, so they should be given as much power as possible. I now realize that this cultural conception of a nerd actually has very little to do with how smart or well-intentioned you really are. The rationalists aren't very good at technical matters (experts in an area can easily spot their errors), but they pull off this culture very well.

Recently, I watched a talk by Scott, where he mentioned an anecdote when he was at OpenAI. Ilya Sutskever asked him to come up with a formal, mathematical definition to describe if "an AI loves humanity". That actually pissed me off. I thought, can we even define if a human loves humanity? Yeah, surely all the literature, art, and music in the world is unnecessary now, we've got a definition right here!

If there's one thing I've learned from all this, it's that actions speak louder than any number of 10,000 word blog posts. Perhaps the rationalists could stop their theorycrafting for once and, you know, look at what Sam Altman and friends are actually doing.