Two ways.

First, sex chromosomes. In mammals, sex is determined by the sex chromosomes - males have XY, females XX. If interbreeding was equal between the sexes of both species, this would be reflected in the frequency of neanderthal genes on each chromosome in the current human population, but it's more heavily skewed toward the Y chromosome than we'd expect if equal pairing was true. This suggests a higher proportion of successful male neanderthal/female human offspring.

Second, mitochondrial DNA. While genomic DNA in a sexually-reproducing species is a mix between the parents, in most mammals the inherited mitochondrial DNA (mtDNA) is purely maternal. This is because only the egg's mitochondria typically survive, though on rare occasion paternal mitochondria are also passed on. There is no known existent neanderthal mtDNA in the human population. This suggests either female neanderthal/male human crosses didn't happen much and/or didn't often produce offspring capable of further reproduction.

Of course, there are many other explanations for all of these. These are just amongst the simplest possible options, and in population genetics, it's not uncommon that the simplest answers are frequently correct.