My experience mirrors yours massively except I ended up married -- well, haha, civilly partnered, although converting that to a marriage is expected to become possible here soon -- to a woman. But for the 9 years before I met her, I was in a long-term straight relationship and boy-howdy did I get some side-eyes for explaining that I was *also* into women and oh god did the gay community deal out a lot of shit at me for being with a man how dare I. At university before that I was informed that I was not allowed to be LGBT rep for my campus because I had a boyfriend at the time. I always *hated* being accused of having passing privilege but coming out was this wall of WE DON'T TAKE YOUR SEXUALITY SERIOUSLY from both sides. So, yes, I am right there with you on all of this, and you have my sympathies.

(Nowadays I spend a lot of time carefully explaining that hey actually I still like men, and that it doesn't matter that I never intend to be with anyone except my wife ever again, my actual sexual identity STILL MATTERS and no stating it to folks is not a come-on. I have a massive and very-visible-to-people-who-know-me-at-all crush on a male celebrity and that's turned out to be more useful than I'd have thought for clarifying my identity to others. But yes, it is still a huge relief to be starting from the lesbian end of the spectrum as an assumption, instead of the other end.)