Yeah it's definitely derogatory in that way, though not a direct insult. I first encountered it while researching cockney rhyming slang. Not sure how common the term is nowadays, but let's just say the older living gens of East Londoners had a certain way with words.