Im glad you mentioned that, because that brings us to the next level of your flawed analysis.
When you spent that extra dollar on that house, earlier than required by the loan, what did that “buy” you? It bought you more ownership of something that appreciated, say, 3-4% per year?
I obtained the same appreciation without spending that dollar. That’s my win #1.
Furthermore, I took that extra dollar that you used to buy more equity in the house and instead I invested it in a bond and/or an equity which, based on my last 20 years or so, earned me 7%. That’s my win #2.
See how simple that is? It’s not that you’re dumb, you’re not. You’re (a) overly concerned about the other side of the transaction (I like win/win), (b) overly complicated in your thinking and (c) fearful.