From what I've seen- if the hands/arms are tight to the body, then no call. If the hands move forward to block, then it's a handling call.
Also- if there is a chance to move out of the way or use your chest/head safely, but the arms come up in defense instead, then it's a handling call.
But it's completely up to the ref to decide.
I realize that proximity to the ball is a consideration, I'm thinking greater than point blank but not farther than it would be to reasonably play the with head or chest.