Liberal Party members who participated in the first removal of a sitting federal MP in Victoria in a generation said there was a moment when they realised Kevin Andrews was toast.

Sounding frustrated and defensive as he sought to keep his privileged position, Andrews cited the heavy responsibility of being defence minister in 2015, when Australian soldiers were fighting in Afghanistan, according to party sources.