I always see rolling a 20 or a 1 as an opportunity for rediculousness to ensue and the modifiers help decide what kind of rediculousness. Skilled swordsman rolls a 1? They have a hilarious fumble meanwhile someone who's never picked up a sword might be stabbing themselves with disadvantage (because the goal isn't to kill the player but to let the dice add flavor. Also accidentally stabbing yourself would probably do less damage than intentionally stabbing someone)

Person who's never picked up a sword rolls a 20? Guess they're now demonstrating awe-inspiring sword skill that they will never be able to match