I'd say that "Professional Youth Sports" is the big scam.
AAU Basketball, Year Round Football, Club Volleyball, Youth Speed and Strength Gyms, Private "Lessons" for all sports.
Kids used to find out what they might be good at by messing around on the playground or in PE class. And we had school sports leagues in elementary school to let kids try out sports without having to be an "elite" player in 4th grade.
We've taken what used to be sports that kids "played" and made it into a business for coaches to make a solid living off of.
Soccer isn't the most expensive. Actually, most kids activities now cost about the same. Between $30 and $50 per hour for some sort of "professionally coached" activity - Sports, singing, art, music, etc.
The answer is really quite simple and is linked to the observation you make that it's not just about soccer, or even team sports.
1) colleges, particularly the top sports and academic colleges, have been engaged in an arms race over the last 20 years...whereas before kids would just go to their local state schools the ease of travel has opened up other possibility....add to that the growing import of students from overseas (where their own programs are capped due to their merit based tracking system) and you have higher competition at the top levels in sports and academia. Whereas my grades were good enough 25 years ago to get me into an Ivy League school, now days with the same scores I'd be laughed out of the interviews. The increased competition has trickled down the ladder so it's no longer enough for kids to just be mediocre and why so many kids apply to colleges these days seem to have founded even their own charities. US soccer, unlike other countries, is college driven, which is why it's no longer good enough just to be a mediocre rec or high school player.
2) opportunities have shrunk, particularly for the middle class, and have increasingly been congregated in several key cities. That all leads to parents being anxious, and feeling that it's necessary to shell out money to give their kids an edge.
3) a generational shift is underway. The Millenials were all about self-actualization and everyone being a winner. The pendulum is swinging the other way. The parents of Gen Z are much more about winning and losing. In school, the kids are tested and ranked. The kids also become something of a status symbol, to be shown off on social media.
4) with parents working and social mores becoming more nervous about dangers that can happen to kids (e.g. kidnappings, being locked in cars), real or perceived, there are no more free ranged children, and in fact having free ranged children might get you locked up by the government. There are no more kids wandering around the neighborhoods ready to play pickup games of football. If you want athletics, they have to be organized. With birthrates falling, there are also fewer of them.
5) US sports have been organized to select kids to play in college. Other nations don't have our collegiate teams. Therefore the focus is on developing good players, but not necessarily just elite players. It's in part why AYSO has fallen into disfavor (because it doesn't supply the kids the necessary training) and why we develop so few homegrown superstars to play in the MLS.
The rest is just the market responding to demand.....