Opposition Leader Anthony Albanese has moved to expel controversial union boss John Setka from the Labor Party but stopped short of refusing the union's money.

The head of the Construction, Forestry, Maritime, Mining and Energy Union's  powerful Victorian branch refused calls to step down on Tuesday for allegedly denigrating domestic violence campaigner Rosie Batty and harassing a woman, despite losing the backing of Labor and the Australian Council of Trade Unions.