Dave, you are correct, the is some effect - it seems "like a lot" when you are holding the wheel with you hands and using your arms as the major force mechanism.
I think when you put the entire weight of the bike and your body into the equation there is probably more negligible effect of the...