I would argue that what's going on is that they are compressing information. And it just so happens that the most compact way to represent a generative system (like mathematical relations for instance) is to model their generative structure. For instance, it's much more efficient to represent addition by figuring out how to add two numbers, than by memorizing all possible combinations of numbers and their sum. So implicit in compression is the need to discover generalizations. But, the network has limited capacity and limited "looping power", and it doesn't really know what a number is, so it has to figure all this out by example and as a result will often come to approximate versions of these generalizations. Thus, it will often appear to be intelligent until it encounters something that doesn't quite fit whatever approximation it came up with and will suddenly get something wrong that seems outside the pattern that you thought it understood, because it's hard to predict what it's captured at a very deep level and what it only has surface concepts of.

In other words, I think it is "kind of" thinking, if thinking can be considered a kind of computation, but it doesn't always capture concepts completely because it's not quite good enough at generalizing what it's learned, but it's just good enough to appear really smart within a certain distribution of inputs.

Which, in a way, isn't so different from us, but is maybe not the same as how we learn and naturally integrate information.

I suspect it's rooted in defensive reactions. People are worried about their jobs, and after being raised to believe that human thought is special and unique they're worried that that "specialness" and "uniqueness" might be threatened. So they form very strong opinions that these things are nothing to worry about.

I'm not really sure what to do other than just keep pointing out what information we do have about this stuff. It works, so in the end it'll be used regardless of hurt feelings. It would be better if we get ready for that sooner rather than later, though, and denial is going to delay that.

Can you give examples of that?

That wasn't the engineers of GPT-4, it was Microsoft who have been fanning the hype pretty heavily to recoup their investment and push their own Bing integration and then opened their "study" with:

“We acknowledge that this approach is somewhat subjective and informal, and that it may not satisfy the rigorous standards of scientific evaluation.”

An actual AI researcher (Maarten Sap) regarding this statement:

The ‘Sparks of A.G.I.’ is an example of some of these big companies co-opting the research paper format into P.R. pitches. They literally acknowledge in their paper’s introduction that their approach is subjective and informal and may not satisfy the rigorous standards of scientific evaluation.

It's PR by Microsoft. I am beginning to doubt the intelligence of many humans rather than that of ChatGPT considering these kinds of comments.

“op” you are referring to is… well… myself, Since you didn’t comprehend that from the posts above, my reading comprehension might not be the issue here.

I don't care. It doesn't matter, so I didn't check. Your reading comprehension is still, in fact, the issue, since you didn't understand that the "learned" vs "programmed" distinction I had referred to is completely relevant to your post.

It’s wether it can grasp concepts from the content of the words it absorbs as it it’s learning data.

That's what learning is. The fact that it can construct syntactically and semantically correct, relevant responses in perfect English means that it has a highly developed inner model of many things we would consider to be abstract concepts (like the syntax of the English language).

If it would grasp concepts (like rules in algebra), it could reproduce them everytime it gets confronted with a similar problem

This is wrong. It is obvious and irrefutable that it models sophisticated approximations of abstract concepts. Humans are literally no different. Humans who consider themselves to understand a concept can obviously misunderstand some aspect of the concept in some contexts. The fact that these models are not as robust as that of a human's doesn't mean what you're saying it means.

the only thing it does is chain words together by stochastic calculation.

This is a meaningless point, you're thinking at the wrong level of abstraction. This argument is equivalent to "a computer cannot convey meaningful information to a human because it simply activates and deactivates bits according to simple rules." Your statement about an implementation detail says literally nothing about the emergent behavior we're talking about.