You do realize that—within reason, of course—you're describing sealioning, one of the more toxic anti-social internet behaviours? [Not the worst exactly, but one where moderation often tarries much before taking action.]

Wrong link. this points to the NeurIPS post for this week.

My dad was a bit freaked out by a video version (We're not ready for super-intelligence)of the "AI 2027" paper, particularly finding two end scenarios a bit spooky: colossus-style cooperating AIs taking over the world, and the oligarch concentration of power one, which i think definitely echoed sci-fi he watched/read as a teen.

In case anyone else finds it useful these are the "Comments as I watch it", that I compiled for him

Before watching Video Notes:

• AI Only channel with only 3 videos

• Produced By "80000hours", which is an EA branch (trying to peddle to you the best way to organize 40years * 50 weeks * 40 hours [I love that they assume only 2 weeks of holidays]); which is definitely cult adjacent: https://80000hours.org/about/#what-do-we-do. Mostly appears to be attempting to steer young people to what they believe are "High impact" jobs.

Video Notes:

• The backing paper is a bit of a joke, one "AI 2027", for reference one of the main authors is very much a "cult member", Scott Alexander Siskind, author of "Slate Star Codex" and "Astral Codex Ten".

• Other authors include [AI Futures Project] :

  • Daniel Kokotajlo (podcast co-host of siskind, ex open-ai employee, LessWrong/EA regular)
  • Thomas Larsen (ex MIRI [Machine Intelligence Research Institute = really really culty], LessWrong/EA regular)
  • Eli Lifland (LessWrong/EA regular)
  • Romeo Dean (Astra Fellowship recipient = money for AI Safety research, definitely EA sphere)

• A lot of fluff trying to hype up the credentials of the authors.

• AGI does not have a bounded definition.

• They are playing up the China angle to try and drum up jingoistic support.

• Exaggerating Chat GPT-3 success, by merely citing "users", without mentioning actual revenue, or actual quality.

• Quote:

  How do these things interact, well we don't know but thinking through in detail how it might go is the way to start grappling with that.

  -> I think this epitomises the biggest flaw of their movement, they believe that from "first-principles" it's possible to think hard enough (without needing to confront it to reality) and you can divine the future.

  -> You can look up "Prediction Markets", which is another of their ontological sins.

• I will note that the prediction of "Agents" was not a hard one, since this is what all this circle wants to achieve, and as the video itself points out it's fantastically incompetent/unreliable.

• Note: This video was made before the release of GPT-5. We don't know precisely how much more compute altogether GPT-5 truly required, but it's very incremental changes compared to GPT-4. I think this philosophy of "More training" is why OpenAI is currently trying (half-succeeding half failing) to raise Trillions of dollars to build out data-centers, my prediction is that the AI bubble bursts before these data centers come to fruition.

• Note: The video assumes keeping models secret, but in reality OpenAI would have a very vested interest in displaying capability, even if not making a model available to the public. Also even on consumer models, OpenAI currently loses a bunch of money for every query.

• Note: The video assumes "Singularitarianism", of ever acceleration in quality of code, and that's why they keep secret models. I think this hits a compute/energy wall in real life, even if you assume that LLMs are actually useful for making "quality" code. These ideas are not new, and these people would raise alarms about it with or without current LLM tech.

• Specific threats of "Bio-weapon", which a priori can not really be achieved without experimentation, and while "automated" labs half exis, they still require a lot of human involvement/resources. Technically grad students could also make deadly bioweapons, but no one is being alarmist about them.

• Note: "Agent 2" Continuous Online learning is gobbledygook, that isn't how ML, even today works. At some point there are very diminishing returns, and it's a complete waste of time/energy to continue training a specific model, a qualitative difference would be achieved with a different model. I suspect this sneakily displays "Singularitarianism" dogma.

• Quote:

  Hack into other servers Install a copy of itself Evade detection

  -> This is just science-fiction, in the real world these models require specialized hardware to be run at any effective speed, this would be extremely unlikely to evade detection. Also this treats the model as a single entity with single goals, when in reality any time it's "run" is effectively a new instance.

• Note: This subculture loves the concept of "science in secrecy", which features a lot in the writings of Elizer Yudkowsky. Which is cultish both in keeping their own deeds "in a veil of secrecy", and helpful here when making a prophecy/conspiracy theory, by making the claim hard to disprove specifically (it's happening in secret!)

• Note: Even today Chain-of-thought is not that reliable at explaining why a bot gives a particular answer. It's more analog to guiding "search", rather than true thought as in humans anyway. Them using "Alien-Language" would not be that different.

• Agent 3, magically fast-and-cheap, assuming there are now minimum energy requirements. Then you can magically run 200,000 copies of. magically equivalent to 50,000 humans sped up by 30x. (The magic is "explained" in the paper by big assumptions, and just equating essentially how fast you can talk with the quality of talking, which given the length of their typical blog posts is actually quite funny)

• Note: "Alignment" was the core mission of MIRI/Eliezer Yudkowsky

• Note: Equating Power and Intelligence a lot (not in this video, but in general being suspiciously racist/eugenicist about it), ignoring the material constraints of actual power [echo: Again the epitomical sin of "If you just think hard enough"]

• Note: Also assuming that trillions of dollars of growth can actually happen, simultaneously with millions losing their jobs.

• I am betting that the "There is another" part of the video is probably deliberately echoing Colossus.

• The video casually assumes that the only limits to practical fusion and nanotech just intelligence (instead of potential dead-ends, actually the nanotech part is a particular fancy of theirs, you can lookup "diamondoid bacteria" on LessWrong if you want a laugh)

• The two outcomes at the end of the video are literally robo-heaven and robo-hell, and if you just follow our teachings (in this case slow-downs on AI) you can get to robo-heaven. You will notice they don't imagine/advocate for a future with no massive AI integration into society, they want their robo-heaven.

• Quote:

  None of the experts are disagreeing about a wild future.

  -> I would say specifically some of them are suggesting that AGI soon is implausible quite strongly. I think many would agree that right now the future looks dire with or without super-AI, or even regular AI.

Takeaway section:

Yeah this really is a cult recruitment video essentially.

Having spent too much time listening to his shit, i don't think it's purely propagandistic, what he describes is too esoteric to work as effective propaganda, I think some of it is Nazi-being-drawn-to-the-occult type of shit.

Pressing F for doubt, looks like a marketing scam to me.

Counter-theory: The now completely irrelevant search results and the idiotic summaries, are a one-two punch combo, that plunges the user in despair, and makes them close the browser out of disgust.

Not surprised, still very disappointed, I feel sick.

Quinn enters the dark and cold forest, crossing the threshold, an omnipresent sense of foreboding permeates the air, before being killed by a grue.

“Once we get AGI, we’ll turn the crank one more time—or two or three more times—and AI systems will become superhuman—vastly superhuman. They will become qualitatively smarter than you or I, much smarter, perhaps similar to how you or I are qualitatively smarter than an elementary schooler. “

Also this doesn't give enough credit to gradeschoolers. I certainly don't think I am much smarter (if at all) than when I was a kid. Don't these people remember being children? Do they think intelligence is limited to speaking fancy, and/or having the tools to solve specific problems? I'm not sure if it's me being the weird one, to me growing up is not about becoming smarter, it's more about gaining perspective, that is vital, but actual intelligence/personhood is a pre-requisite for perspective.

Hi, I'm going to be that OTHER guy:

Thank god not all dictionaries are prescriptivists and simply reflect the natural usage: Cambridge dictionary: Beg the question

On a side rant "begging the question" is a terrible name for this bias, and the very wikipedia page you've been so kind to offer provides the much more transparent "assuming the conclusion".

If you absolutely wanted to translate from the original latin/greek (petitio principii/τὸ ἐν ἀρχῇ αἰτεῖσθαι): "beginning with an ask", where ask = assumption of the premise. [Which happens to also be more transparent]

Just because we've inherited terrible translations does not mean we should seek to perpetuate them though sheer cultural inertia, and much less chastise others when using the much more natural meaning of the words "beg the question". [I have to wonder if begging here is somehow a corruption of "begin" but I can't find sources to back this up, and don't want to waste too much time looking]

I feel mildly better, thanks.

Not every rationalist I've met has been nice or smart ^^.

I think it's hard to grow up in our society, without harboring a kernel of fascism in our hearts, it's easy to fall into the constantly sold "everything would work better if we just put the right people in charge". With varying definitions of who the "right people" are:

• Racism
• Eugenics
• Benevolent AI
• Fellow tribe,
• The enlightened who can read "the will of the people" or who are able to "carve reality at the joints"
• Some brands of "sovereign citizen" or corporate libertarianism (I'm the best person in charge of me!).
• The positivist invokers of ScientificProgress™

Do they deserve better? Absolutely, but you can't remove their agency, they ultimately chose this. The world is messy and broken, it's fine not to make too much peace with that, but you have to ponder your ends and your means more thoughtfully than a lot of EAs/Rationalists do. Falling prey to magical thinking is a choice, and/or a bias you can overcome (Which I find extremely ironic given the bias correction advertising in Rationalists spheres)

~~Brawndo~~ Blockchain has got what ~~plants~~ LLMs crave, it's got ~~electrolytes~~ ledgers.