Height is about 80% genetics, 20% environment (nutrition and general health). Grandparent's genetics are a factor since that is where the parents inherited their genes.

A child inherits two copies of every gene from their parents, one copy from each. Your family could be like mine, where my grandfather was quite tall and my grandmother was unusually short, suggesting my grandfather had double pairs of genes for height and my grandmother had double pairs of genes for... not height. Their children, my mother and her siblings, then had an even assortment of tall and short genes.

Her brother married a woman whose parents followed the same tall father/short mother pattern and they had three sons: two of about the same height as the parents (even tall/short gene mix) and one who is about 20 cm taller than the rest (few or no short genes).