Perhaps I should have been more clear. I didn't mean it's a direct substitution for "from where", just that that's its meaning. A grammatical translation of the sentence would be "go back to where you came from".