My understanding is that yes, it will hurt theirs (at first) but not as much as it'll help yours, as their verified carries more weight than your projected. Also, in time assuming you move up, the impact will change.
Say you win in a close match, 6-4 6-3. Their rating immediately drops to 4.3...