Labor believes the political backlash against its plans to end cash refunds for excess franking credits will be limited because the majority of the almost one million people affected already vote conservative or live in Liberal seats.

Labor's relationship with self-funded retirees plumbed new depths on Wednesday when shadow treasurer Chris Bowen told those upset with the policy that they were free to take their vote elsewhere.