If I block a user, shouldn’t their votes not apply?

I’m guessing the answer is “it’s too hard to make that work” but I figured I’d ask.