54
Nick Bostrom Has a Plan for Humanity’s ‘Big Retirement (www.wired.com)
submitted 1 week ago by sanitation@lemmy.radio to c/technology@lemmy.world
33 comments fedilink hide all child comments
top 33 comments
sorted by: hot top controversial new old
[-] justsomeguy@lemmy.world 27 points 1 week ago

His 2014 book Superintelligence was an early examination of AI’s existential risk. One memorable thought experiment: An AI tasked with making paper clips winds up destroying humanity because all those resource-needy people are an impediment to paper clip production.

Good thing we have this philosopher to have the most superficial thoughts about AI while he poops. His second book now seems to be along the lines of "guys, AI will fix everything". What a great follow up to AI will destroy everything. Top twist.

[-] Iconoclast@feddit.uk 15 points 1 week ago

The paperclip maximizer is a thought experiment. That's all. It's an overly simplistic way to explain the gist of a more complex idea. The fact that even this basic thought experiment goes over people's heads just further proves why that simplification was needed in the first place.

[-] EvergreenGuru@lemmy.world 7 points 1 week ago

Your comment would be more convincing if you laid out the complex idea you’re alluding to, instead of saying that a simple example is all people need.

As far as I can tell, thought scientists stay losing, because pretending your thoughts comprise a form of science that ends in a measurable result is sophistry.

[-] Iconoclast@feddit.uk 15 points 1 week ago

It's to illustrate the alignment problem. What you literally ask isn't always what you actually want. This is usually obvious to humans but not necessarily to an AI. If you sit in a self-driving car and tell it to take you to the airport as fast as possible, you might arrive three minutes later covered in vomit with the entire police department after you. That's obviously not what you wanted, but you got exactly what you asked for.

The paperclip maximizer is a cartoon example of this. If you just ask it to make as many paperclips as possible, that becomes its priority number one and everything gets turned into paperclips and you might not get the chance to tell it this isn't what you meant.

A kind of real-life example is the story of a city that started paying people for rat tails to eradicate the rat population, only for folks to start breeding rats instead to make money. It's a classic case of unintended results due to unspecific requirements.

[-] MangoCats@feddit.it 2 points 1 week ago

the story of a city that started paying people for rat tails to eradicate the rat population, only for folks to start breeding rats instead to make money.

Or the real life story of the US elementary school students who saved up money to buy and then free slaves, which - when examined closer - was found to be driving growth in the slave trade, not slowing it down.

In both cases - you figure out what's off kilter, and you stop doing that.

It's a lot easier to turn off "AI machines" than, for instance, powerful industries like Oil and gas...

[-] eleitl@lemmy.zip 2 points 1 week ago

Alignment is undecidable, so no point wasting synapseseconds.

[-] Iconoclast@feddit.uk 4 points 1 week ago

It's not a matter to decide but a problem to try and solve. In most cases we get to learn from our mistakes but when it comes to AGI we might not.

Or are you suggesting we shouldn't even think about it but rather just roll the dice and see what happens?

[-] eleitl@lemmy.zip 3 points 1 week ago

Undecidable in the sense that no solution can exist for that problem class. You can start with the definition of what exactly you're aligning with, how you measure that, how you derive applicable system evolution constraints from your measurements, and just what humanity is, in the iterative context.

Apart from that we're already in an out of control winner-takes-all arms race where AI is used by competing nations, including social control and battlefield. Ivory tower is a meal ticket with no practical relevance.

[-] MangoCats@feddit.it 1 points 1 week ago

Ivory tower is a meal ticket with no practical relevance.

See also: https://rmst202.sites.olt.ubc.ca/files/2022/04/illich_deschooling-society.pdf

[-] MangoCats@feddit.it 1 points 1 week ago

you might not get the chance to tell it this isn’t what you meant.

And that is where the thought experiment left the tracks - lifted off with escape velocity and is now passing Voyager 2...

In what cartoon world do we not get a chance to shut off the Doomsday Device? I mean, it was a funny little twist at the end of Dr. Strangelove, but as realistic as many elements of that story were, that was not one of them.

[-] Iconoclast@feddit.uk 1 points 1 week ago* (last edited 1 week ago)

I don't think you fully appreciate the implications of creating something orders of magnitude more intelligent than us. You can't outsmart something smarter than you. Even if it was only as smart as the smartest human, being a computer it would still process information a million times faster. Everything would happen in super-slow motion from its perspective. It would have so much time to consider each move.

Humans aren't anywhere near the strongest primate on Earth, yet we're by far the dominant one. I don't think a gorilla has any idea just how much smarter we are, and even if it did, it would probably still assume that a war with humans would mean us outnumbering them, hitting, biting, and throwing things at them. They'd have no clue we can end them from a distance without them ever knowing what hit them. They can't even imagine all the ways we could - and have - screw things up for them, even when we have nothing against gorillas.

The point isn't that I think this is absolutely going to happen, but just to highlight that we're effectively rolling the dice on it and seeing what happens - which I find incredibly irresponsible. This whole "it'll be fine, we can always turn it off" attitude is incredibly naive and short-sighted.

[-] MangoCats@feddit.it 1 points 1 week ago

You can’t outsmart something smarter than you.

And, yet, we have rich idiots making all our top level decisions. https://www.cnbc.com/2014/07/16/icahn-too-many-companies-run-by-morons.html

I don’t think a gorilla has any idea just how much smarter we are

I don't think most people have any idea just how smart a gorilla, or dolphin, or squid, or pig, or any of thousands of other species are.

it would probably still assume that a war with humans would mean us outnumbering them, hitting, biting, and throwing things at them.

Many people, but not all, are very rigid in their thinking. Similarly, some animals are adaptable: https://theconversation.com/city-animals-act-in-the-same-brazen-ways-around-the-world-279977

They’d have no clue we can end them from a distance without them ever knowing what hit them.

Many hunted animals have evolved a fear of humans at a distance. All the megafauna of Africa remaining today are only there because they evolved alongside humans, instead of being blindsided and hunted to extinction before they figured out what we can do.

Will we be blindsided by our computers (any more than we already have been)? Undoubtedly. Will they turn around and start eating us because they're so fast and smart? Probably not.

we’re effectively rolling the dice on it and seeing what happens - which I find incredibly irresponsible.

Yep. Pretty much like developing the fossil fuel industry, or cutting all the mature trees off the face of three continents, hunting whales to near extinction, killing all the megafauna in the Europe and the Americas, desertification of the cradle of civilization through unsustainable farming, etc. etc.

I agree, it's irresponsible. I disagree with those who liken it to a world war IV global apocalypse in a millisecond singularity.

[-] idiomaddict@lemmy.world 2 points 1 week ago

The “experiment” is one you conduct on yourself, it’s not for thinking about a process and using your imagined results as the basis of further study. It’s very useful in a number of non scientific fields, and it can serve as an aid in scientific education though, so it shouldn’t be written off generally.

The paper clip thought experiment is a punchy, memorable example of the conflict between what input you give to a computer and what the computer interprets from that. The goal is for people who hear it to remember that they need to be thoughtful about what exactly they want and precise in their phrasing when they’re programming or training an AI.

[-] MangoCats@feddit.it 2 points 1 week ago

The paper clip thought experiment is a punchy, memorable example

See also: childrens' books about the dangers of magic. https://gutenberg.ca/ebooks/eagere-halfmagic/eagere-halfmagic-00-e.html

[-] MonkderVierte@lemmy.zip 2 points 1 week ago

Actually, that's neuroscience.

[-] Shanmugha@lemmy.world 26 points 1 week ago

One good thing AI bubble did: it showed me we have so many rich idiots that it is an actual problem

[-] BloodMuffin@lemmy.ca 10 points 1 week ago

I couldn't finish the article. what a nincompoop.

[-] Arrandee@lemmy.world 9 points 1 week ago

Nick Bostrom takes himself waaaaaaaayyy too seriously.

[-] jafra@slrpnk.net 8 points 1 week ago

Are AI and AGI the same now? Is there a new theory of "just has to be big enough"? That would explain americas self-destructive planning of datacenters.

I for one would immediately switch on an AGI, i think even a 20% probability for a benevolent AGI is acceptable, compared with what humanity is doing.

[-] Iconoclast@feddit.uk 6 points 1 week ago

AGI is always AI, but AI isn't always generally intelligent. AI is the parent category that AGI is a subcategory of. It's like the difference between the terms "plant" and "dandelion." All dandelions are plants, but not all plants are dandelions.

[-] jafra@slrpnk.net 2 points 1 week ago

You missunderstood what i adked. I know very well the difference. What i don't get is why promoting stupidAIs will "solve all problems".

[-] Iconoclast@feddit.uk 1 points 1 week ago* (last edited 1 week ago)

AGI is capable to solve all our problems. It's not LLMs that Bostrom is talking about here.

[-] jafra@slrpnk.net 1 points 1 week ago* (last edited 1 week ago)

And that was my question. Are AGI now amy more real than a year ago? Or is this narrative just just big moneys wet dream and helpful in growing public acceptance of stupidAIs.

[-] Iconoclast@feddit.uk 1 points 1 week ago

AGI is purely theoretical at this point. Nobody has a truly generally intelligent AI system.

[-] MangoCats@feddit.it 1 points 1 week ago* (last edited 1 week ago)

Early examples of AI came out in the 1960s, things that could solve algebra equations, give basic pschological interviews... They were "smart" in very limited scopes.

[-] zqps@sh.itjust.works 2 points 1 week ago

Any new technology is subject to the same problems under capitalism, specifically maximising profits to the detriment of anything else. This is especially bad with centralised tools. An AGI wouldn't just magically take global control.

[-] Sxan@piefed.zip 2 points 1 week ago

An AGI wouldn’t just magically take global control.

We can only hope. A true AGI would see þe harm of þe current wealþ distribution. Wiþ any luck it'd take over an redistribute it.

[-] zqps@sh.itjust.works 1 points 1 week ago

You really believe that with Elon Musk and Peter Thiel in charge of its initial parameters and training, bar any oversight? That stretches hope too far in my book.

[-] Sxan@piefed.zip 1 points 1 week ago

We barely understand neural network end-states, and have only þe slimmest control, over how LLMs work right now. If we do achieve AGI, I doubt þey'll have much control. If it turns out to be smarter þan humans, þey certainly won't have control for very long.

[-] Rekall_Incorporated@piefed.social 7 points 1 week ago

For some reason this reminds me of the "effective altruism" movement (if you can call it a movement).

[-] Gsus4@mander.xyz 5 points 1 week ago

Oh no, not this fiend again...

[-] W98BSoD@lemmy.dbzer0.com 4 points 1 week ago

http://archive.today/vPchv

[-] Malyca@lemmy.zip 1 points 1 week ago

That would involve a lot of trust in the AI and it's training materials

this post was submitted on 12 May 2026
54 points (100.0% liked)

Technology

84831 readers
3761 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws