You aren't wrong: for most of human history infant mortality and lack of vaccines for childhood diseases pulled down the average, but if you made it to adulthood you could expect to live a normal length of time. IDK what the life expectancy was back then specifically (Tolkien undoubtedly saw huge improvements in medicine and pharmacology during his life), but germ theory had already been well established by the time Tolkien was born and they had antibiotics. So, that's not the whole story in this case.

Tolkien fought in World War I when he was in his 20s, and an absurd number of his friends and peers died. That war (and the flu, which broke out towards the end of the conflict and almost certainly spread more quickly and widely thanks to the close quarters and movement of military forces) had a huge impact on demographics.

Tolkien is in the solid red line in the graph below:

(Edit: the British military was also doing this thing called "pals battalions", where you'd serve alongside people from your town. Which was allegedly good for morale, but meant that the people you saw die were people you'd grown up with and that one bad day of fighting could wipe out an entire town's military-aged male population.)