Well not having sexual attraction at all would be asexual, rather than demi-sexual, and even then, you can still find or think someone is attractive without actually being sexually attracted to them, as it's not really the only form of attraction to exist.

And unfortunately, usually due to any number of personal problems, including but not limited to lack of impulse control, it seems quite difficult for many to not want sex with whoever they find attractive, as I've personally seen several times by now.

But of course that's hard for me to exactly understand given I don't seem to feel sexual or romantic attraction to anyone myself, despite having or being close enough emotionally to people that either would be reasonably apparent.

And of course, I don't mean offense toward people in consensual circumstances such as an open relationship with that second line, quite the contrary, as that's not an impulse control issue, nor an problem if consent is given and done safely, etc.