Basically, yeah.

Eventually the distant galaxies will fade into memory as their light is lost to the expansion of the universe, then the stars will all burn out, the radioactive materials will all decay, and black holes evaporate in the final flashes of light to illuminate the universe.

With no sources of heat left, what remains gets down near absolute zero as residual energy is lost as infrared radiation. The heat death of the universe is the long, cold dark at the end of everything.