Hi John,
I can relate to your situation somewhat. I'm male, mid 40's, married (2nd time), have always liked women and their bodies, but had some interest in guys too on & off since my youth. Suppressed those feelings till my 30's, and then explored some of the curiosity (oral with another guy, give and receive).
I've never been into cross-dressing or gender confusion, but do have an interest in guys' packages sometimes. I'll notice them in porn moves, and sometimes at the pools in Vegas if something/someone really stands out (good looking bartender, or super well-endowed guy in a Speedo!).
I've been with guys maybe 5 or 6 times over the last few years, but only 1 receiving anally. Willing to do it more though. I've no interest in being the giver ("on top"), but I have had some fun once I get past the stigma. I guess it's mainly penis-envy or something. I've had small, long, thick, and regular....it's true that size doesn't really matter overall.
There's no clear answer, but only something you have to decide for yourself. My wife would explode if she knew, so I keep it to myself, and am very careful on the few times I've explored "the other side of the fence."