First and foremost, don’t just take his word for it. My fiancé almost broke things off with me when one of his guy friends claimed that he’d fucked me years earlier. I had not (the guy looks like a fucking muppet), and fortunately could verify that at the time when this guy had allegedly “rawed my tight asshole” and “titty fucked me until my face was glazed” I was in fact living in an entirely different country. In actuality, I had merely met the guy who claimed we boned a handful of times and been friendly. But since he’d wanted to fuck me, he thought it would be really cute to brag about a made up conquest to his buddies. Super fun. Almost cost me my relationship. Very happy that my fiancé went to the trouble to ask me about it before he broke things off so I had the chance to tell him I hadn’t and wouldn’t have EVER slept with that guy.
Secondly: Your feelings are valid wether or not they’re completely “reasonable”. Personally in your case I’d ask myself a few questions before ending things with someone I clicked with. Like: why does this bother me so much? In what way and why does this change the way I feel about this person? Does it matter how long ago/the circumstances under which this happened? Was I under the impression that she was a virgin before I had this information? Would I be bummed out if someone I liked broke things off with me because I had a slept with someone they knew? Do any of these answers make me reconsider how I feel?