We were neither investing nor paying down the mortgage early in our case. We were spending the money elsewhere. I believe this is the reality for most people.
The former rule of thumb was to invest your age into these lower-risk assets.
If you leave that money in stocks for ten years or more, you’ll likely have more money than by paying down a mortgage.