Once the Season has ended, the character you made and its progress will be transferred to the Eternal Realm. All items you collected, including in your Seasonal stash, will be transferred too. You can take this character into Sanctuary whenever you please, but Season-specific features such as Malignant Hearts will no longer be accessible.

From the Season introduction page, so it seems that what you'll miss if you don't play will be only the season cosmetics. And the chance to play with new mechanics for the duration of the season.