I agree this is why American in particular are annoyed by the game (and one of the reasons why professionally it has never caught on in the United States to the extent the other big 3 have). Great observation. But it's also been a problem outside of the rest of the world. In Latin America referee abuse is practically part of the game, and from my relatives in Spain and my son's pen pal in England, there are similar issues in those countries that understand the game better than we do.
It requires a special person to be a good referee (given the fluidity in the rules). They have to have a personality confident enough to make the tough calls (instead of avoiding them), with good judgement and a good understanding of social dynamics to keep the game under control, yet not tyrannical or foul tempered. And that's besides having a firm understanding of the Laws, experience which is only built up by calling the game and making mistakes, having a good vision (that's where I always fall flat), being physically fit, and caring. They also have to be mentally strong enough to handle the tough set of circumstances (winning take all coaches, combative parents, teenaged hormones, a large field, a fast moving ball and ambiguous rules) they are handed. I have great admiration for those that are able to do it and be good CRs, because they have a very unique talent set, and we aren't paying them enough to attract such good talent (we shouldn't be surprised by a lot of the bad we get, particularly at the lower levels, as a result).
In one game my son played in a few weeks back we had a really great referee that knew how to manage the game, both avoiding calling every foul but also counting the ones that mattered both for game speed and game tone. I made a point afterwards of introducing myself and told him as a sometimes baby ref, I admired his style, even if the parents at both ends were chipping. They had some reason to chip: he called the handball rule the old way...but otherwise a great ref.