You don't say explicitly what the coworker said to you regarding you being lazy, but that is specifically what you need to address. I would go to your manager to make her aware of the situation, "xx told me he thinks I'm lazy, but I put in more hours and do more work than he does. I'm going to address this with him directly, but wanted to put it on your radar first."

When you go talk to him, be super calm the entire time. Literally no anger, no irritated inflection. Recognize you're likely dealing with an adult child. Your job here is going to be to make him so uncomfortable he never bothers you again through relentless questioning. "The other day, when you said you thought I was lazy, I realized I got to work well before you did that morning, and it seems like I had gotten more work done than you too. What were you basing that observation on?" Whatever his reply is, turn around and ask a clarifying question about it. "I'm just trying to understand where you're coming from. When you sit and chat with a coworker for an hour without doing any work, I don't bother you about it, so I'm really wondering why you're bothering me and what that's about." "If we're here to hold each other accountable to do work, I can certainly do that for you too. Do you want to start right away?"

Be super genuine and well-meaning here. He will be utterly confused and will likely realize he's better off never saying anything like that again. You could even report back to your supervisor that you spoke with him and offered a deal to hold each other accountable to productivity standards, but he declined.