Peter Dutton says Liberal MP Julian Leeser was at odds with the overwhelming majority of his party colleagues on the Indigenous Voice to parliament, doubling down on opposition to the referendum as the shadow attorney-general quit to support the Yes campaign from the backbench.

A longtime campaigner for Indigenous constitutional recognition, Mr Leeser resigned from shadow cabinet on Tuesday because he opposed the party’s policy, saying the time for the Voice had come.