Yeah, picking up other people's balls at the course is a major faux pas. If you find a ball in the rough then it's usually okay to take it if there's nobody on the hole you're playing. In your defense, they're not supposed to leave a hole half played, and if they were going to do that, they should have left their ball markers and taken their balls with them.