I'm actually not really in favor of punishing refs for blunders - unless it's obvious match fixing. I'll admit as someone who used to be a ref himself I am biased on that one, but it is just what it is. Blunders will happen, in the heat of the moment when you are forced to make a decision within a few seconds you will misjudge and fuck up things sometimes. And it is also simply part of the football drama we all love.
But I agree on punishing the video assistants: they are not on the ground, not even in the stadium (I don't know how it works in other countries, but in Germany they're always in Cologne), they can easily get a replay of the situation asap, they don't have to deal with players and coaches having a spergout around them, etc...