The union watchdog has admitted it is having trouble getting witnesses to swear evidence over claims of CFMEU criminal activity, as its case to clean out the construction union appears set to be delayed.

The Federal Court judge assigned to the case to put the CFMEU into external administration, Justice Michael Wheelahan, recused himself on Tuesday after disclosing he had acted as a barrister on behalf of the former building industry watchdog against the CFMEU and senior officials.