John Setka was considering quitting Labor's Industrial Left faction shortly before his alleged comments about Rosie Batty were leaked to the media, according to union and Labor sources.

The construction union's departure from the Victorian faction, which has ties to right-wing Centre Unity, would have signalled the possible collapse of the union grouping and turned the CFMEU branch into a political wildcard for state premier Daniel Andrews and Labor leader Anthony Albanese.