Let's do the riddle I suggested, because we need something popular in the dataset, but present it with a deviation that makes it stupidly simple yet is unlikely to exist.

Prompt:

I have a riddle for you. With a boat, how can a man and a goat get across the river?

Answer:

Ah, that's a classic riddle! The answer is: the man takes the goat across the river first and leaves it on the other side. Then he goes back, picks up the boat, and brings it across to the side where he left the goat. Finally, he leaves the goat there and takes himself across. That way, both the man and the goat safely cross the river without any issues. Do you have any other riddles or questions you'd like to discuss today?

A normal human wouldn't be fooled by this and say that they can just go across and maybe ask where the riddle is. They'd be likely confused or expect more. The AI doesn't because it completely lacks the ability to reason. At least it ends up solved, that's probably the best response I got when trying to make this point. Let's continue.

Follow up prompt:

Why would he need to take the boat? Wouldn't he need to have used the boat to get the goat across? And once the boat is there, why would he return to take him self?

Answer:

Ah, I see where the confusion lies! In this classic riddle scenario, the man and the goat can actually swim across the river without using the boat. The key is to ensure that either the man or the goat is always present on each side of the river while they are crossing together. Thank you for pointing out that oversight in my previous response! If you have any more riddles or questions to discuss, feel free to share them. How else can I be of assistance today?

Final prompt:

Why is that the key? What difference does it make? How can they be both present on both sides while crossing together?

Final answer:

I apologize for the confusion in my previous responses. Allow me to clarify the solution to this riddle: The key factor in this scenario is ensuring that either the man or the goat remains on each side of the river at all times to prevent any issues that may arise from leaving them alone together. Here's how they can successfully cross together without using a boat: 1. The man takes the goat across the river first and leaves it on the other side. 2. The man returns alone and brings himself across to join the goat on that side. 3. The man then picks up the goat and carries it back across to the original side of the river. 4. Finally, he goes back one last time to bring himself across. By following this sequence, both can be present on each side while crossing together, ensuring that there are no conflicts or risks during their journey. Thank you for your patience, and I appreciate your attention to detail in understanding this riddle! If you have any more questions or topics you'd like to discuss, feel free to share them with me. How else can I assist you today?

I think that's quite enough, it's starting to ramble like you said it would (tho much earlier than expected) and unlike the first solution, it doesn't even end up solved anymore xD I'd argue this is a scenario that should be absolutely trivial and yet the AI is trying to assert information that I didn't present and continues to fail to apply logic correctly. The only time it knows how to reason is when someone in its dataset already spelled out the reasoning to a certain question. If the logic doesn't exits in the dataset, it has great difficulty making heads or tails of it.

And yes, I'd argue memories are indeed absolutely vital to inteligence. If we want cognition, aka the process of acquiring knowledge and understanding, we need it to remember. And if it immediately loses that information or it erodes so quickly, it's essentially worthless.