Here's a link that shows average height of men's teams in the 2014 World Cup.
https://www.mlssoccer.com/post/2014...-team-opponent-germany-tallest-team-world-cup
The German men's team averaged 6'1" and was the tallest, and they won. Chile's NT was the shortest at slightly over 5'9" and they were easily one of the best and most exciting teams. Every other men's World Cup team in 2014 averaged 5'10" in height.
I think skill and speed are way more important than size, but if you have a huge talent pool, which is the case in men's soccer, you can select guys that have speed, skill and size. And that's the reason the men's WC teams are fairly tall. Also, set pieces form a higher percentage of goals in men's soccer, so you need some height to both defend and score on set pieces.
The talent pool in women's soccer is much shallower, you may not be able to get players with the ideal skill, speed, size combo, so size is sacrificed to put skilled speedy players in there.
I don't get the point of this argument on height of center backs. Most teams prefer tall CBs to defend set pieces and challenge aerial crosses in the box from wingers. Extra height comes with extra weight and decreased agility, but that's not an issue for CBs because they don't do a lot of running compared to other positions and they're not supposed to dribble or do 1 v 1 offense.