The more of the Cosmere you read, the more things will connect and the clearer the picture will be. Every time I re-read SA, I see new connections I'd missed before. Some of that is familiarity with the magics of other planets, because that changes your read from "character did a weird thing" to "hey! that's magic from this other planet; why/how has it turned up here?"
In the earlier books, the crossovers between worlds/magics and the underlying "how things work" are more subtle and you'll miss things on first read. In more recent books, it's more overt.
Some of that is because of how much the protagonists themselves understand. For example, in the first Mistborn trilogy the characters really don't understand what's going on on their own planet, so of course you don't get a good explanation. In Secret History, the POV character does run into people who know quite a lot about what is going on, so when Secret History revisits the events of the main trilogy you're able to understand the forces driving those catastrophic events.
The characters in SA started off thinking magic wasn't real and knowing nothing about realms and worlds beyond their own. They are learning a lot through their spren and Hoid, but there is still a lot that they don't know. And you as the reader are learning along with them.
I've never had a scarecrow wrecked. Can that actually happen?