Common & classic abuser. Abusers are so good at this and you aren't the only who has had to go through this before. Abusers are good manipulators and love to twist things around on us to make us look or feel bad. They don't want to look like the bad one because heaven forbid it won't look good on them, so they twist everything around to make us look like we are in the wrong. This is just another common, classic, abuser. I swear, I'd love to gather them all up and get a bonkin' bat to knock sense into every single one of them. But they are all so dense it wouldn't phase them at all.
I think your abuser does this on purpose and he knows which buttons to push on you. He knows how to work you in other words. When he knows you are in town and wants to see the child, he will push your buttons by saying things like you just mentioned, "I think you tell our daughter bad things about me......" and he is doing that on purpose because he knows it makes you mad. Regardless if you are or not, he knows how to push your buttons because he still wants to control you and he still wants that power over you. IMO, if he is so danged concerned about yall's daughter, he'd get into counseling. But he won't because he doesn't think anything is wrong w/him. As usual, another common classic. He won't change and he won't ever get it. You, my friend, have done nothing wrong. He, on the other hand, is just an abusive, self centered, jerk, who will never change. Abusers are just impossible to deal with.
Keep venting as much as you want. We are here for you.