I wonder if people realize that rules are generally thought up as a "lesson" or response to some negative experience. I think the idea of sex "too" early really depends on the level of expectation a person (usually the woman) allows themself to have after having sex early on in the dating process.
I had sex with my boyfriend on our first date. I was enjoying the moment, got into it, and as an adult decided this was what I wanted to do, eyes wide open, AND have no expectation that it would turn into a relationship afterwards. I think that is the key. We continued having sex, both of us having a great time and yet being we were raised more or less the same way, believing that this kind of behavior isn't the means to a solid foundation for a relationship. I still don't believe it necessarily leads to solid relationships in general, having sex right away with someone does not a relationship make.
But what happened afterwards is what turned it around into a relationship. We were honest with each other, accepted each other, we had a good time together outside the bedroom as you seem to be doing with your guy, and, there was no expectation on either end that just because we'd had sex early on, that instantly cemented us to one another into a relationship. The relationship came through relating over time, as it should.
Truth be told, I did have a talk with him a few weeks after we met and told him that while I did not regret having sex with him right away, if I had the choice to make over again, I would choose not to have had sex with him so soon and get to know him first. THAT is when we stopped just fooling around and started to actually date. That was over four years ago and we're happier than ever.
As long as you hurt no one, including yourself, live your life by your own "rules." You're the only one looking back at you in the mirror.
Best of luck,