It's normal to feel nervous: he's seen you naked! :P
He's a nice guy that you had a connection with in the past. Sure, it got out of hand and led to sex before you meant it to and you panicked, but it's been 4 years- you can just say you panicked and weren't sure what to do at the time. Apologize if you feel it will help YOU to feel better.
As for EG, you don't miss him. You miss what you SHOULD have had with him, not what you actually did.
Go and have FUN on your date, and try not to think of it too much as a date.