It's not much different than if you were to go to a business that buys and sells gold. They'll have the current market rate on display and for video games eBay is a decent place to look for the market rate. With retro games there may at least be a bit of lag if they have prices directly on the games and not do something shady like looking up current prices at checkout.

What you said is definitely a problem though. The person selling a game should look at recent sold copies over the past 6ish months and consider the physical quality of the game in question to determine a price better; I don't think a lot of shops put much time or thought into this aspect of it.

I don't really know how you could do it any other way when retro games are a finite resource that isn't being produced any more.