IMO........No time is too soon and no time is too long. Some people have sex on the first date, some not until they're married, some anywhere in-between. Each individual has to decide when the right time is for them to have sex with someone. I had sex with my husband on our 3rd date, some guys I dated I didn't have sex with at all, some I engaged in oral sex (giving & receiving), manual stimulation but not IC. Each date, each relationship, I made different choices based on what was right for me at that time.
It doesn't matter what your friends do, it doesn't matter what anyone else does, you need to determine for yourself when it's okay for you. My rule of thumb for everything is, if in doubt, don't. If you wonder whether or not you should have sex with him then you might be better off waiting. When it's the right time for you, you won't have to question it.
Ask yourself if you have sex with him and the two of you break-up down the road, will you regret having sex? Are you wanting to have sex out of fear that if you don't he'll break up with you? If you have sex, have it because it's what you want to do but not with the expectations that sex will guarantee a future together.