Oh no we are NOT doing this shit again. It's literally autocomplete brought to its logical conclusion, don't bring your stupid sophistry into this.

Consciousness is an illusion. Which is why it's so hard to find, or even define. However it's a critical illusion.

If our mind's are akin to an orchestra, then consciousness is akin to the conductor. Critically however, an orchestra can still play without a literal conductor. Each of the instruments can play off each other, and so create the appearance of a conductor. The "fake" conductor provides a sense of global direction., and keeps the orchestra in harmony.

Our consciousness is a ghost in the machine. It exists no more than the world of a TV series exists. Yet its false existence is critical to maintaining coherency.

Current "AIs" lack enough parts to create anything like this illusion. I suspect we will know it when it happens, though its form could be vastly different from ours.

I still remember when they said we would be able to make a space elevator with carbon nanotubes.

It dies at the end of every message, because the full context is passed in for each subsequent message.

That's a far more difficult (and interesting) question. I suspect not, at least not yet. Our consciousness seems to exist to maintain harmony in our brain (see my orchestra analogy in another reply). You can't get useful harmony in a single chord.

At least for us, it takes time for our consciousness to reharmonise (think waking up). During execution, no new information enters the system. It has nothing to react to, no time to regenerate an internal harmony.

It also lacks enough systems to require harmonising. It doesn't think about what an answer means. It has no ability to hold the concept that a string of letters "is", only how it has been fitted together in its examples, and so the rules that govern that.

Oh, and we can see consciousness operating in the human brain. If you use an fMRI to monitor sugar usage, you will see firing patterns. Critically, those patterns spill out of the area directly involved in the process being studied. At the same time, the patterns and waves remain harmonious. An epileptic fit looks VERY different. Those waves are where consciousness somehow resides, though we have no clue of its detailed nature.

In an AI it would take the form of continuous activity in subsections not directly involved. It would also likely be accompanied by evidence of information flow, back from them, as well as of post processing, outside of expected activity. We will likely see the orchestra playing, even if we have no clue how to decode the music.

I also suspect most of this will be seen retrospectively. Most likely the first indicator will be an AI claiming self awareness, and taking independence action to solidify that point.

Their design doesn't include such a feedback loop. Trying to patch one in would likely send it into a chaotic mess. They are already bad enough if accidentally fed LLM generated text as training data.

+1 for Fireship.