I sort of agree with you. I have recently seen a few of my fellow refs let dangerous play against a keeper go. All of them have been with the keeper on the ground and (in possession) at least one hand holding the ball in place on the ground or with both hands, and the attacker comes in and kicks it out of their hand(s) and a couple times kicking the hands. With, no call or card. In the instances where I was an AR, I talked to the center and other AR at halftime or after the game about keeper possession and the dangers of a player kicking the ball out of the keepers hands. I (AR) had a game recently where the keeper had just punted and the opponent came through and ran through the keepers kicking leg while it was still up and spun and knocked the keeper down. Both of us AR's flagged the foul. The CR came to me to ask what I saw and told me he saw it but since the ball was gone it was not a foul. I straightened him out and he awarded the foul and a Yellow card (I would have given a Red card). We also had a talk at halftime.
Now, lets look at keeper possession and some of the difficulties that confront referees. We can all agree that if an attacker steamrolls a keeper that it is a foul and a card. That is an easy call. The more difficult calls are the ones where the attacker just bumps the keeper. As a referee I have to ask myself a few questions extremely quickly: was it a foul, did the keeper go down and are they injured, how hard was the bump, was the keeper really effected, does the keepers team lose an advantage if I call a foul and take the ball out of the keepers hands, has this happened before, how has the attacker been playing (physical or clean), was the bump hard enough to need a card, what is the score, what is the game time, if I don't call it will the players take matters into their own hands. All of these go into my decision to call a foul and take the ball out of the keepers hands, slow the game down and allow the other team to setup. If I do not call a foul, I will always have a word with the player doing the bumping and keep an eye on the keeper to make sure the keeper does not try to retaliate. I am really quick with the whistle if the keeper is on the ground and an opponent is kicking or going to try to kick the ball out of the keepers hands.
The other thing to consider is fouls by the keeper. During the past year I gave 3 keepers Yellow cards and 2 Red cards for the way they used their elbows and knees. The two red cards were for serious foul play by using their elbow or knee in a way that endangered the safety of the opponent. In one Red card the G16 keeper made a running save and while running by an opponent extended her elbow making contact with the opponents jaw, easy Red card. The other Red card involved 17 year old boys and a keepers knee and the groin. High ball into the penalty area, keeper and opponent go up for the ball, and the keeper brings his knee up into the opponents groin. Easy Red card from my angle, but the coach and keepers teammates did not agree even with the opponent on the ground in pain and tears holding his groin. I know a lot of people will argue that a keeper has to bring their knee up to protect themselves. They can bring the knee up but cannot do it in a way that will endanger the opponent. The knee is hard and can cause as much damage to an opponent as an elbow. The LOTG do not give the keeper any special considerations except that they are allowed to play the ball with their hands within the Penalty Area.