When I read this question my first though was YUCK, of course not! However....
I am an only child but I did get hand me downs from a cousin when $$$ was tight in my family and I was younger, and would be excited to see what would be in the bag of "new to me" clothes. Though I admit other kids did make fun of my clothes when I was around 8-10 (especially for wearing "floods"), and I do think it gave me a bit of a complex (probably why I am such a clothes addict today!). But I never had hand me down underwear or swimsuits and think I would not have wanted it.
However, if the kids don't have a problems with it, especially if they are very young and it is from a sibling, I don't see anything wrong with it as long as they are washed in hot water, not frayed, and bleached, and unstained. Hand me downs save money and are better for the environment. We are such a throwaway society. And for big families...well I just don't know how people afford to have 5 or 6 kids. You gotta do what you gotta do I guess. However, psychologicaly, I don't think a parent should ever force a child to wear hand me down underwear if they have any issues with it/negative feelings about it. It's not worth giving a child a complex over, it's not that expensive to buy new socks and undies. But if they don't know the difference and don't care, why not?