Well, you finally said it. "There are clear physical performance ability differences between men and women". There are also clear performance ability differences between MTF on suppressants or post surgery (no testosterone), FTM (no male puberty), and intersexed in comparison to xy males. The reality is they cannot compete either with an xy male on active testosterone, testosterone being a performance enhancing drug. So why do xx women get special treatment from you but not anyone else? Why do xx women get the benefit of this luxury, all the while you (and allies) are complaining that it is also unfair for xx women to compete with a MTF who has gone through puberty?
There are only a handful of possible rationales. It's not clarity because clarity is best served by everyone competing in the same category...you are still saying xx women are entitled to special treatment through a league of their own. One possibility is we don't care about them. Another possibility is they aren't worthy of respect. A third possibility is they did it to themselves so they deserve it. Otherwise, I'm struggling to see why one group of people deserves special treatment, but another does not. (Fair warning I do see a fourth possibility, but it's a dangerous one because it opens the door to acknowledging women's sport as "lesser" and therefore things like equal access to scholarships goes out the window).
As to why we should care, it's easy, because we are all individuals, granted by God with individual rights, entitled to equal treatment before the law. Just because it's a hard case doesn't mean we get to ignore it. Our liberty is made up of the hard cases.