I'm giving a lot of benefit of the doubt, because the Internet is a poor medium for talking about personal stuff, and meanings are not always clear. So I'll assume nothing, and start from scratch.
There is no shame in having some threshhold of attractiveness for yourself, as you consider new people as potential dates, and as you go on first dates. Most everyone has a "type" of some kind. It's perfectly fine to enjoy a person's company, but not be "into" them. It's not unusual to want to hang out with someone new and interesting, who it turns out you're not all that attracted to. And there's nothing wrong with needing a date or two in order to suss out your own feelings toward someone, and how well you click. It's actually a sensible dating policy to give a little time, and avoid snap judgements.
Guys get a lot of weird messages from society, and little to no advice that isn't laced with innuendo and assumptions. So issue one is to clarify in your own mind if you actually "fancy" this woman, or maybe you just like her as a friend.
Issue two is, assuming you are into her, then clearly she's attractive to you. That's what should matter. If you like her, and the only hesitation you've got about whether this is a good idea is whether your friends will also think she's attractive, then the problem and the solution are really simple: you need to grow up. You're not dating your friends. Putting their opinions and preferences above your own is stupid and immature, and it can't possibly make you happy.
