From the comments

The average person concerned with existential risk from AGI might assume "safety" means working to reduce the likelihood that we all die. They would be disheartened to learn that many "AI Safety" researchers are instead focused on making sure contemporary LLMs behave appropriately.

"average person" is doing a lot of work here. I suspect the vast amount of truly "average people" are in fact concerned that LLMs will reproduce Nazi swill at an exponential scale more than that they may actually be Robot Hitler.

Turns out if you spend all your time navelgazing and inventing your own terms, the real world will ignore you and use terms people outside your bubble use.

When you make your alarmist arguments dishonestly, how can I freak out about the end of the world?

Let me translate one point:

People are already talking about an AI rights movement in major national papers

A PhD student got an opinion piece published on the hill dot com.

Basing your security on Movie plot threats is a pretty bad idea.

E: also somewhere in the NSA there is a junior analyst feverishly writing a report right now:

We should stop developing AI, we should collect and destroy the hardware and we should destroy the chip fab supply chain that allows humans to experiment with AI at the exaflop scale. Since that supply chain is only in two major countries (US and China), this isn't necessarily impossible to coordinate

(somebody on LW highlighted the last line as important (I also think the line is wrong but im not going to 'somebody is wrong on the internet' help the wannabe terrorist cultists)).

we simply don't know how the world will look if there are a trillion or a quadrillion superhumanly smart AIs demanding rights

I feel like this scenario depends on a lot of assumptions about the processing speed and energy/resource usage of AIs. A trillion is a big number. Notably there's currently only about 0.8% this number of humans, who are much more energy efficient than AIs.

We should also consider how the efforts of AI can be directed towards solving human aging; if aging is solved then everyone's time preference will go down a lot and we can take our time planning a path to a stable and safe human-primacy post-singularity world.

This shit is so funny. It always comes back around to immortality with these people.

I spent my adult life to this point seeking to understand enlightenment and transcendence, and if you aren't reading the doctrines of an immortality cult (lots of that in daoism and tescreal, oddly enough) then mostly you come to find that overcoming the fear of death is a big part of it. You can't have a free and open mind if the shadow of your mortality looms, so you learn to let go of it as one more attachment.

I think it's very revealing of what shallow minds these people truly have that the desperate craving for immortality is so naked in their beliefs. That the concept of it goes so unexamined as well (oh we just make everyone immortal and then we all agree to debate and solve our AI problem? Because that would work?)

Like, how stupid do you have to be to say on the one hand that unexamined AI risk requires a massive effort to run simulations and game theory out the consequences, but then on the other hand to be like "things would be better if everyone was immortal. I will take no further questions on that."