Non-binary seems like it could have several non-compatible meanings, so I wanted to list some of those meanings and see if there are any others out there I don't know.
One way I could think of non-binary is as being a kind of third gender category, like there are men, women, and non-binary people. In this sense of non-binary a butch woman who considers themselves a woman would not be non-binary because they are a woman.
Sometimes non-binary is used like "genderqueer" is sometimes used, as a generic description of anyone who doesn't fit perfectly in the narrow confines of the binary genders (i.e. men and women). In this sense a butch woman could see themselves as a woman, but also as genderqueer and non-binary, as they do not conform to binary gender norms for women.
Another way non-binary seems to be used (related to genderqueer in its historical context) is as a political term, an identity taken up by otherwise cis-sexual and even cis-gendered people who wish to resist binary gender norms and policing. In this sense even a femme cis-sexual woman might identify as non-binary. Sometimes this political identity label might come with a gender expression that cuts against the gender expectations for the assigned sex at birth, but it doesn't have to. (I recently met two people whose gender expressions matched their assigned sex at birth but who identified as non-binary in this political sense.)
I was wondering what other meanings of non-binary are out there, and how they are commonly used.
Note: gatekeeping what is "really" non-binary seems pointless to me, since I agree with Wittgenstein that "language is use".
I know people get heated about policing what a word means (and I am guilty of this myself), but in the interest of inclusion, pluralism, and general cooperation in our community I think we can find a way to communicate with overlapping and different meanings of a shared term.
Interesting notions, it sounds like a lot of your notion of "non-binary" is similar to the second meaning I had in mind, basically the idea of being genderqueer or gender non-conforming.
I can really relate to some of what you are saying about not feeling awful being nominally gendered male by others, I had a sort of indifference to gender and felt like if people wanted to call me he/him that it was on them, not me. They were gendering me, so I didn't have to take it that seriously.
When I watched the Transition Channel videos, especially the Common Excuses to Transitioning video, I realized I might be more trans than I had previously considered (I thought of myself as nominally non-binary and "gender non-conforming" before, not really thinking of those terms as being trans per se), and most importantly that I might have been suppressing myself and ignoring dysphoria. It was a good coping strategy while growing up since transitioning was never going to be an option then, and it was unsafe not to conform to the assigned masculine gender.
Like Natalie Wynn and Mia Violet describe, I too found living as a boy not too bad. I didn't really have thoughts that I was a girl before puberty hit. Even living as a man wasn't that distressing as I could just ignore gender and dissociate to cope.
Anyway, your description of your gender reminds me a lot of myself before I transitioned (not that this means you will be like me, I just can relate to what you are describing). It's hard for me because transitioning is so difficult, there is still such a strong desire in me to ignore this problem and not prioritize it. I feel selfish for prioritizing it, and I also feel like it's a huge risk for something that I have been able to live without for so long.
That said, HRT changed the balance - I'm not transitioning so I can live as a woman primarily, I'm transitioning because repressing and having androgen dominance turned out to be impacting my mental health in ways I didn't know until I tried HRT.